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EXECUTIVE  SUMMARY 


OBJECTIVE 

Space  and  Naval  Warfare  Systems  Center,  San  Diego’s  (SSC  San  Diego’s)  Command  21  project 
is  directed  at  supporting  the  needs  of  senior  decision  makers  and  support  staff  in  military  command 
centers.  As  part  of  that  effort,  a  prototype  wall-sized  shared  display — or  “Knowledge  Wall”  (K- 
Wall) — that  fused  mission-relevant  information  was  created  to  support  shared  situation  awareness, 
to  facilitate  group  interaction,  and  to  augment  the  decision-making  capabilities  of  senior  staff.  The 
prototype  K-Wall  was  designed  to  meet  14  user  requirements  that  we  had  identified  with  a  previous 
cognitive  task  analysis  of  potential  K-Wall  users.  Prototype  K-Walls  were  implemented  ashore  in  the 
Joint  Command  Center  of  the  Naval  War  College  (NWC)  and  in  the  Command  21  Laboratory  of 
SSC  San  Diego,  and  onboard  the  USS  Coronado  (AGF  1 1)  for  the  Global  2000  War  Game. 

The  Global  War  Game  provides  a  venue  for  debuting  potential  high  technology  solutions  for 
future  naval  needs.  As  such.  Global  2000  afforded  an  invaluable  opportunity  to  observe  the  prototype 
K-Wall  undergoing  extensive  real-time  usage  in  an  operationally  realistic  setting  by  exactly  the  sort 
of  experienced  real  Navy  users  for  which  it  was  intended. 

To  facilitate  a  usage,  utility,  and  usability  evaluation  of  the  K-Wall,  a  research  plan  was  developed 
for  gathering  both  observational  and  automated  data  in  a  minimally  invasive  fashion  during  the 
game.  This  evaluation  was  focused  on  (a)  the  extent  to  which  the  design  solutions  met  the  14  user 
requirements  and  whether  there  were  new  requirements  that  future  versions  of  the  K-Wall  should 
meet,  (b)  whether  the  design  of  the  information  products  (summary  pages)  supported  the  communi¬ 
cation  of  mission-relevant  information  that  could  be  easily  fused  to  provide  an  integrated  operational 
picture,  and  (c)  whether  there  were  usability,  ergonomic,  or  other  issues  that  need  to  be  addressed  by 
the  Command  21  program  for  future  success  of  the  K-Wall  and  its  satellite  associated  technologies. 

RESULTS 

We  found  that: 

•  The  prototype  K-Wall  adequately  supported  most  of  the  14  user  requirements.  However,  these 
were  met  with  varying  degrees  of  success. 

•  The  K-Wall  eradicated  the  need  for  a  traditional  8-hour  briefing  cycle;  instead,  the  K-Wall  was 
used  continuously  for  situation  assessment. 

•  The  “summary  pages”  shown  on  the  K-Wall  provided  users  high-level  summarized  views  of 
the  operational  picture  and  a  means  to  easily  navigate  through  the  “Knowledge  Web”  to  get 
more  detailed  information.  However,  users  had  difficulty  determining  which  content  was  new. 

•  The  support  applications  used  to  produce  pages  and  graphical  content  to  populate  them  (Tac- 
Graph  and  SumMaker)  were  a  success.  However,  information  providers  had  difficulty  tracking 
whether  users  had  viewed  their  information  products. 

•  K-Wall  users  found  the  K-Wall  intuitive  and  easy  to  use  and  they  preferred  it  to  other  informa¬ 
tion  tools  available  to  them.  However,  some  usability  issues  related  to  interaction  with  the 
Wall  and  visibility  of  information  on  the  Wall  were  identified. 


RECOMMENDATIONS 

The  data  provided  important  feedback,  pointing  to  the  specific  needs  that  must  be  met  by  future 
Command  21  research  and  development  efforts.  Specifically,  we  recommend  future  K-Wall  designs 
need  to  support  improved: 

•  Tools  to  support  attention  management  and  change  detection,  including  tools  to  support 
navigation  to  changed  pages  in  the  Knowledge  Web. 

•  Tools  and  revised  business  processes  to  support  improved  multi-tiered  collaboration,  including 
feedback  and  guidance  for  information  providers  on  content  access  of  their  pages. 

•  Design  layouts  to  support  improved  text  legibility. 

•  Information  integration  across  functional  areas. 

•  Tools  and  methods  to  communicate  information  age  and  source  on  the  summary  pages. 
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1.  INTRODUCTION 


1.1.  PREVIOUS  RESEARCH  AND  COMMAND  CENTER  NEEDS 

The  U.S.  Navy  is  increasingly  conducting  missions  outside  the  bounds  of  traditional  warfare. 

These  missions  are  often  conducted  with  fewer  personnel  than  in  the  past,  and  with  command  and 
control  that  is  more  distributed  in  nature.  Such  mission  characteristics  do  not  always  fit  well  with  the 
methods,  technologies,  and  compatibilities  of  current  command,  control,  communications,  com¬ 
puters,  and  intelligence  (C'‘l)  systems.  Improvements  in  display  systems,  information  management 
tools,  and  human-computer  interfaces  (HCI)  are  needed  to  support  decision  processes,  improve 
situation  awareness  (SA)  (Endsley,  1995),  improve  combat  team  interactions,  and  enable  the  tactical 
team  to  recognize  critical  events  and  operate  within  the  rules  of  engagement.  The  Command  21  pro¬ 
ject  of  the  Space  and  Naval  Warfare  Systems  Center,  San  Diego  (SSC  San  Diego)  is  working  toward 
providing  such  improvements. 

Previous  interviews  with  Joint  Operation  Center  (JOC)  senior  staff  (specifically.  Battle  Watch 
Captains  [BWCs])  established  a  general  need  for  tools  to  support  shared  SA  and  decision-making  in 
the  JOC  (Miller  &  Klein,  1998;  Moore  &  Averett,  1999).  These  analyses  identified  the  existing  Con¬ 
cept  of  Operations  (CONOPS)  for  the  JOC  and  revealed  an  important  need  for  improved  displays  and 
information  management  systems.  In  the  current  CONOPS,  information  products  that  are  generated 
by  anchor  desk  personnel  for  senior  staff  are  distributed  continuously.  These  products  also  form  the 
basis  for  formal  briefs  that  are  given  at  fixed  intervals  (typically,  three  times  a  day)  for  the  Admiral. 
Between  briefs,  BWCs  and  Assistant  Battle  Watch  Captains  (ABWCs)  monitor  events  primarily  by 
verbally  interacting  with  anchor  desk  personnel  and  others  outside  the  JOC  to  remain  constantly 
apprised  of  the  operational  situation.  Moore  and  Averett  uncovered  a  critical  JOC  need  for  improved 
technologies  to  support  BWCs  in  maintaining  SA  and  for  continually  visualizing  the  “big  picture.”  In 
structured  interviews,  BWCs  reported  finding  the  tasks  of  gathering,  fusing,  and  disseminating 
operational  information  both  time-consuming  and  error-prone.  BWCs  need  better  ways  of  acquiring 
SA  rapidly  and  then  disseminating  that  awareness  to  support  shared  SA  and  to  facilitate  decision¬ 
making  for  senior  staff.  What  is  needed  is  a  way  of  visualizing  mission-status  in  a  continuous  fashion 
that  does  away  with  the  8-hour  briefing  cycle  altogether. 

1.2.  KNOWLEDGE  WALL  CONCEPT  AND  DESIGN  FOR  GLOBAL  2000 

SSC  San  Diego  has  been  developing  the  “Knowledge  Wall”  (K-Wall)  concept  as  a  solution  to 
these  needs  (see  Figure  1).  The  K-Wall  is  a  wall-sized  shared  display  consisting  of  contiguous  dis¬ 
play  windows  that  bring  together  information  from  multiple  sources.  The  idea  behind  the  K-Wall  is 
that  providing  a  processed  and  fused  presentation  of  the  “information  space”  on  a  single  large  display 
should  support  cognitive  processes  such  as  data  integration,  pattern  recognition,  event  memory,  and 
distributed  cognition  amongst  its  users.  The  K-Wall  is  intended  to  be  large  enough  to  provide  a 
shared  operational  picture  to  multiple  personnel  to  promote  shared  SA  and  to  provide  focus  for  col¬ 
laboration  among  its  users. 

In  order  to  determine  the  specific  K-Wall  information  and  functionality  requirements  for  a  Navy 
JOC,  a  series  of  structured  interviews  were  conducted  by  SSC  San  Diego  subject  matter  experts 
(Smallman,  et  al.,  2001).  The  participants  in  the  interviews  were  senior  level  personnel  (Admirals, 
BWCs,  ABWCs)  and  JOC  anchor  desk  personnel  (J2,  J3,  J6,  JAG,  etc.)  from  elements  of  Third  Fleet, 
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Figure  1.  The  K-Wall  integrates  mission-relevant  information  from  a  variety  of  sources. 

Command  Carrier  Group  1  (CCGl),  and  Command  Carrier  Group  3  (CCG3)  who  planned  to  partici¬ 
pate  in  the  Global  2000  War  Game.  The  interviews  focused  on  the  K-Wall  in  the  context  of  its 
operational  use  during  a  mission.  Analysis  of  the  responses  led  to  the  identification  of  14  critical  user 
requirements  that  the  K-Wall  must  meet.  These  14  requirements  could  be  categorized  into  the  Gen¬ 
eral,  Format,  Content,  and  Feature  requirements  for  an  initial  K-Wall  prototype.  The  requirements 
and  the  design  solutions  implemented  in  the  initial  K-Wall  prototype  are  shown  in  Table  1. 

To  provide  knowledge  management  in  a  Web-centric  fashion,  the  K-Wall  is  an  html-based  tool 
that  can  be  roughly  conceptualized  as  a  multi-screen  Web  browser.  To  provide  shared  SA,  the  K- 
Wall  is  a  shared  display.  Of  course,  how  well  simply  sharing  information  promotes  shared  SA  is 
obviously  an  open  question.  In  fact,  several  recent  empirical  studies  addressing  this  question  have 
found  mixed  results  (Farley  et  al.,  1998;  Bolstad  &  Endsley,  1999).  However,  given  the  time  con¬ 
straints  of  the  prototype  development  effort,  it  was  decided  that  this  was  the  most  obvious  first  solu¬ 
tion  to  the  requirement.  To  provide  integrated  information,  the  K-Wall  displays  the  status  of  the  vari¬ 
ous  functional  areas,  whose  anchor  desks  are  providing  mission-relevant  information  in  the  form  of 
summary  pages.  To  provide  an  intuitive  graphical  interface,  the  K-Wall  displays  graphical  infor¬ 
mation  where  possible;  some  of  these  graphics  were  generated  by  a  tactical  graphics  (TacGraph) 
support  application  created  specially  for  the  purpose  (Bank  &  Moore,  2000).  To  support  a  consistent 
format  for  its  users,  the  K-Wall  features  summary  pages  that  were  developed  using  a  support  appli¬ 
cation  called  SumMaker  created  specially  for  the  purpose  (Averett  &  Moore,  2000).  To  support  a 
tactical  focus,  the  K-Wall  allows  tactical  displays  such  as  Command  and  Control  PC  (C2PC)  tactical 
software  (currently  used  by  U.S.  Marine  Corps  and  other  U.S.  military  organizations)  to  be  displayed. 
To  show  supplemental  information,  the  K-Wall  features  summary  pages.  To  display  mission  goals 
and  objectives,  this  initial  K-Wall  prototype  can  simply  display  a  text  document  listing  those.  Anchor 
desk  information  products  may  be  displayed  using  links  provided  on  the  summary  pages.  Collabora¬ 
tion  is  supported  by  enabling  collaborative  tools  (such  as  InfoWorkspace,  NetMeeting,  etc.)  to  be  run 
in  a  window  on  the  K-Wall.  In  this  first  prototype  wall,  cognitive  support  is  limited  to  displaying  the 
outputs  from  a  limited  number  of  nonintegrated  cognitive  support  tools.  The  need  for  a  flexible 
configuration  is  supported  by  providing  a  control  screen  that  allows  any  page  to  be  viewable  in  any 
display,  and  to  make  visible  a  host  of  commercial  off-the-shelf  (COTS)  and  government  off-the-shelf 
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(GOTS)  software  tools.  To  support  drill-down,  the  K-Wall  is  populated  with  html-based  summary 
pages  that  are  the  first  level  in  a  hierarchy  of  information  that  can  be  accessed  via  links  to  more 
detailed  information.  There  is  no  current  display  of  information  age  and  reliability  on  the  K-Wall 
apart  from  text-flags  that  indicate  the  last  time  a  given  page  was  modified.  To  support  tactical  over¬ 
lays,  the  K-Wall  supports  the  presentation  of  TacGraph- generated  augmented  and  modified  tactical 
displays  (Bank  &  Moore,  2000).  It  was  this  initial  K-Wall  design  that  was  implemented  and  then 
evaluated  at  the  Global  2000  War  Game. 


Table  1.  The  14  K-Wall  user  requirements  and  the  design  capabilities  provided  to  meet 
them  in  the  prototype  K-Wall  for  Global  2000. 


User  Requirement 

K-Wall  Protot^e  Design  Capability 

Shared  SA 

Shared  display 

Integrated  Information 

Co-located  summary  pages 

Intuitive  Graphical  Interface 

Graphical  presentation  when  possible 

rormai 

Consistency 

Consistent  format  summary  pages 

Tactical  Focus 

Ability  to  view  multiple  tactical  displays 

Supplemental  Information 

Summary  pages  on  peripheral  displays 

Mission  goals  and  objectives 

Text  document 

Content  Anchor  Desk  Output 

Summary  pages  with  links  to  more  info 

Connectivity/Collaboration 

Collaboration  tools  (IWS) 

Cognitive  Support 

Limited  output  from  nonintegrated  cognitive 
support  tools 

Flexible  Configuration 

Any  pages  viewable  in  any  display 

Drill-Down 

Multiple  scalable  views,  links  to  more  info 

reaiure 

Information  Age  and  Reliability 

- 

Tactical  Overlays 

Various  software  for  tactical  graphic  presentation 

1.3.  THE  K-WALL  AT  GLOBAL  2000 
1.3.1.  The  Global  2000  War  Game 

The  Global  2000  War  Game  was  played  over  a  period  of  2  weeks  (14-25  August  2000).  Of  the 
10  working  days,  it  included  1  day  of  training,  6  days  of  operational  play  and  IVi  days  of  post- game 
Executive  Sessions.  The  game  consisted  of  three  operational  phases:  a  pre-hostilities  phase  (Phase  1), 
a  hostilities  phase  (Phase  2),  and  a  post-hostilities  phase  (Phase  3).  The  command  structure  consisted 
of  a  Commander  Joint  Task  Force  (CJTF),  a  Commander  in  Chief  (CINC),  and  a  National  Command 
Authority  (NCA)  supported  by  the  component  commanders  and  personnel  assigned  to  one  of  13 
functional  areas’.  The  K-Wall  prototype  provided  the  operational  picture  to  the  CJTF  and  CINC  staff 


‘  The  thirteen  functional  areas  designated  for  use  during  Global  2000:  Air  Defense  (AD),  Deep  Strike  (DS),  Effects 
(Eff),  Fires  Coordination  (FC),  Ground  Control/Closed  Air  Support  (GC),  Intelligence  Surveillance  and  Reconnais¬ 
sance  (ISR),  Information  Warfare  (IW),  Logistics  (Log),  Operational  Maneuver  From  the  Sea  (OMFTS),  Special 
Operations  Forces  (SOF),  Sea  Control  (SC),  Rear  Security  (RS),  Theater  Missile  Defense  (TMD). 
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by  displaying  information  products  produced  by  the  anchor  desks  elements  making  up  these  func¬ 
tional  areas.  The  information  products  were  continually  updated  over  time  during  game  play.  In 
practice,  this  meant  that  the  pages  were  updated  multiple  times  every  hour. 

1.3.2.  Initial  K-Wall  Implementation 

Initial  K-Wall  prototypes  were  implemented  in  the  Joint  Command  Center  (ICC)  at  the  NWC 
during  the  Global  2000  game.  One  purpose  of  the  K-Wall  is  to  support  collaboration  and  shared  SA 
across  remote  locations  so  another  implementation  of  the  K-Wall  was  in  the  Crisis  Action  Center 
(CAC)  on  the  USS  Coronado  (AGF  11).  (Another  version  of  the  K-Wall  was  constructed  in  the 
Command  21  Laboratory  at  SSC  San  Diego  for  research  and  development  purposes). 

The  initial  K-Wall  prototype,  shown  in  Figure  2  at  the  NWC,  consisted  of  two  large  focus  moni¬ 
tors  (58.25-inch-diagonal  SmartBoard  rear  projection  displays)  surrounded  by  ten  smaller  (21 -inch) 
peripheral  monitors.  An  additional  21 -inch  monitor  was  included  to  provide  VideoWTC  capabilities. 
By  default,  the  large  focus  monitors  displayed  tactical  information  while  supplemental  information, 
i.e.,  the  summary  page  output  from  the  13  functional  areas,  was  displayed  on  the  smaller  peripheral 
monitors.  The  intent  of  this  design  was  to  provide  the  K-Wall  user  an  intuitive  visual  metaphor  for 
the  role  of  the  supplemental  data,  specifically,  to  embed  the  tactical  data  in  the  context  of  more 
detailed  supplemental  information.  Any  information  on  the  smaller  monitors  could  easily  and  quickly 
be  “brought  into  focus”  on  either  of  the  central  monitors  using  custom-built  K-Wall  HCI  created  for 
the  purpose. 


Figure  2.  The  initial  Global  2000  K-Wall  prototype  implementation  (here  at  the  JCC  at  the  NWC). 

Two  sets  of  mice  and  keyboards  provided  input  to  the  K-Wall.  In  general,  one  of  these  inputs  was 
controlled  by  one  of  the  K-Wall  technologists  and  the  second  was  provided  to  the  K-Wall  users 
(senior  staff).  The  focus  monitors  also  provided  touch  screen  interaction  capabilities. 
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1.3.3.  Business  Processes 

The  K-Wall  was  intended  to  provide  its  users  (senior  staff:  CJTF  at  the  JCC  and  CINC  on  the 
Coronado)  with  a  shared,  integrated  view  of  the  operational  situation  by  displaying  information  from 
multiple  sources  (the  13  functional  areas)  in  a  single  location.  The  basic  K-Wall  system  setup  that 
was  used  during  Global  2000  is  shown  in  Figure  3.  Information  products  were  created  and  continu¬ 
ally  updated  at  the  functional  area  anchor  desks  and  then  stored  in  shared  locations  accessible  to  the 
computers  systems  of  all  Global  2000  players.  Many  of  these  products  (such  as  summary  pages,  see 
below)  were  created  with  the  express  purpose  of  populating  the  K-Wall.  The  K-Wall  users  controlled 
the  K-Wall  content  by  making  requests  to  change  or  move  the  information  displayed  on  the  K-Wall. 


Figure  3.  The  basic  K-Wall  system  setup  as  implemented  during  Global  2000. 

K-Wall  Users.  Figure  4  shows  the  layout  of  the  JCC  during  the  Global  2000  War  Game.  The  posi¬ 
tions  of  the  primary  K-Wall  users  (CJTF,  CoS,  J2,  J3,  J4,  J6,  BWC,  BWl,  BW2),  as  well  as  of  the  K- 
Wall  technologists  and  the  JCC  Knowledge  Manager  (KM),  are  shown  in  blue.  Gray  regions  repre¬ 
sent  horizontal  surfaces  (tables)  and  hatched  regions  represent  computer  systems  (laptop,  computers 
on  the  command  table,  keyboards,  and  monitors).  The  K-Wall  could  display  any  of  the  information 
that  was  accessible  to  game  players,  including  Web  pages  and  other  documents  and  tactical  and  other 
software.  The  K-Wall  users  controlled  changes  in  the  content  and  configuration  of  the  information 
displayed  on  the  K-Wall  by  making  requests  to  the  K-Wall  operator  (a  K-Wall  technologist  or  one 
the  CJTF  staff  dedicated  to  K-Wall  operation)  or  by  taking  control  of  the  wall  themselves. 
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Figure  4.  The  layout  of  the  JCC  at  the  NWC  during  Global  2000  (top-down  view). 

Information  Products  and  Information  Providers.  During  the  game,  summary  pages  and  other 
information  products  from  the  13  functional  areas  were  created  to  be  displayed  on  the  K-Wall  (and 
elsewhere).  An  example  summary  page  is  shown  in  Figure  5.  The  summary  pages  consisted  of  five 
parts: 

1.  Color-coded  status  indicators  for  the  short-term  (Today),  mid-term  (Tomorrow),  and  long¬ 
term  (Long-Range)  time  periods 

2.  Important  alerts  to  new  information  concerning  the  functional  area 

3.  Impacts  and  implications  of  those  alerts 

4.  Related  links  for  more  information 

5.  A  graphic  related  to  the  alerts  or  current  status 

This  consistent  format  was  imposed  to  meet  the  user  requirement  for  consistency  (see  Table  1). 
Consistency  in  the  placement  of  information  on  displays  is  necessary  to  allow  users  to  scan  for 
information  most  efficiently  (see  Mayhew,  1992;  Wickens,  1992).  Color-coded  status  information 
was  a  prominent  feature  of  the  summary  page.  It  was  intended  to  support  rapid  SA  of  the  overall 
mission  status  of  the  entire  functional  area  through  an  intuitive  graphical  format.  The  concept  was  to 
enable  the  K-Wall  user  to  rapidly  acquire  and  integrate  information  from  the  various  functional  areas 
and  to  prioritize  his  or  her  knowledge  retrieval  with  the  K-Wall.  The  alerts,  text,  and  pictures  could  be 
linked  (using  html  hyperlinks)  to  other  information  to  initiate  drill-down  to  more  detailed  information. 
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Figure  5.  Example  summary  page  (left)  and  Overview  page  (right). 

P  The  intent  was  that  certain  aspects  of  the  summary  pages  would  remain  constant  (such  as  the 
functional  area  status  for  today,  tomorrow,  and  long  range,  and  the  fixed  location  of  the  different 
types  of  information)  to  provide  efficient  integration  of  information  across  multiple  summary  pages. 
The  figure  also  shows  an  Overview  page,  which  displayed  the  “worst  case”  status  value  for  each  of 
the  functional  areas.  This  page  was  not  initially  part  of  the  K-Wall  design.  However,  as  the  K-Wall 
concept  prototype  matured,  the  need  for  such  a  page  became  clear  as  there  were  insufficient  monitors 
to  display  a  summary  page  for  each  functional  area  and  numerous  special-purpose  pages.  Thus,  the 
Overview  page  was  intended  to  give  users  an  integrated  summary  across  all  the  functional  areas, 
including  any  that  were  not  presently  being  displayed.  The  K-Wall  technologists  monitored  the 
content  of  the  summary  pages  and  manually  updated  this  Overview  page  over  time. 

rior  to  the  beginning  of  operational  play,  at  least  one  individual  working  in  each  functional  area 
was  chosen  to  be  responsible  for  the  creation  of  the  summary  pages.  These  individuals  (hereafter 
referred  to  as  “information  providers”)  were  provided  with  the  SumMaker  software  (Averett  & 
Moore,  2000),  with  which  they  could  develop  and  update  summary  pages  (see  Figure  6).  This  soft¬ 
ware  allowed  them  to  edit  the  color  of  the  status  alerts,  change  the  text  and  pictures,  and  create 
hyperlinks  from  the  text  and  graphics  to  other  information  products,  such  as  Microsoft  Word 
documents,  briefs,  etc.  They  were  briefly  trained  in  its  use  prior  to  game  play. 

The  information  providers  were  also  given  the  TacGraph  tool  (Bank  &  Moore,  2000),  which 
allowed  them  to  create  interactive  tactical  pictures  that  also  provided  hyperlinks  to  other  documents. 
An  advantage  of  TacGraph  was  that  this  enabled  tactical  graphics  to  be  published  in  html  format  and 
then  later  be  edited,  if  needed  (see  Figure  7).  This  allowed  for  multi-level,  graphics-based  informa¬ 
tion  products  to  be  developed,  and  saved  information  providers  considerable  time  because  they  were 
spared  from  their  slow  and  inefficient  old  business  process  of  creating  single-use,  single-level  custom 
tactical  graphics  with  a  combination  of  nonintegrated  applications  (e.g.,  C2PC  and  PowerPoint). 

They  received  training  on  the  use  of  these  tools  as  well  as  on  the  intended  content  and  usage  of  the 
summary  pages.  Information  providers  were  encouraged  to  make  use  of  this  tool  for  briefing  pur¬ 
poses. 
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Figure  6.  The  SumMaker  interface. 


Figure  7.  Example  output  created  with  the  TacGraph  tool. 

Each  functional  area’s  summary  pages  and  the  information  products  that  they  linked  to  were  stored 
in  shared  folders  so  that  they  could  easily  be  accessed  by  other  game  players.  Access  to  each  of  the 
1 3  functional  area  summary  pages  was  available  through  the  “channels”  (favorites)  menu  on  the  K- 
Wall  or  through  links  in  the  Web-based  War  Game  Information  Grid  System  (WIGS). 
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2.  DATA  COLLECTION 


The  constraints  of  collecting  data  during  a  real-time  operational  exercise  where  players  could  not 
be  interrupted  during  the  execution  of  their  tasks  led  to  the  creation  of  a  carefully  tailored  data  col¬ 
lection  plan. 

2.1.  DATA  COLLECTION  PLAN 

The  data  collection  effort  was  very  focused  in  scope.  It  centered  primarily  on  the  usage  of  the  K- 
Wall  in  the  JCC  at  the  NWC.  Some  observational  data  was  also  collected  for  the  K-Wall  onboard 
Coronado.  One  purpose  of  the  data  collection  was  to  determine  whether  the  K-Wall  design  had  met 
the  design  goals  by  examining  it  in  the  context  of  operational  use  (during  the  Global  2000  War 
Game).  Another  goal  was  to  establish  usage  patterns  and  identify  utility  and  usabihty  issues  related 
to  the  K-Wall.  The  data  collection  plan  had  several  objectives.  It  attempted  to  determine  whether: 

1 .  The  K-Wall  design  solutions  supported  the  14  user  requirements  identified  in  the  previous 
interviews  and  whether  there  were  additional  requirements  that  would  require  new  K-Wall 
design  features  and  content. 

2.  The  information  products  (summary  pages  and  others)  met  the  information  requirements  of 
K-Wall  users.  In  particular,  whether  the  content  and  format  of  the  summary  pages  provided 
the  K-Wall  users  a  consistent,  intuitive  representation  of  the  status  across  the  different  func¬ 
tional  areas  that  could  be  easily  fused  to  provide  an  integrated  picture  of  the  operational 
situation. 

3.  There  were  usability  or  ergonomic  issues  related  to  K-Wall  use  or  operation. 

2.2.  DATA  COLLECTION  TECHNIQUES 

The  data  collection  techniques  employed  during  Global  2000  were  restricted  to  passive  observa¬ 
tion  and  automated  data  collection.  Because  of  the  free  flow  and  continuous  nature  of  the  game,  no 
questionnaires  that  would  disrupt  play  were  given  and  direct  questioning  of  players  was  limited.  The 
K-Wall  application  software  collected  some  data  automatically  while  other  data  were  recorded  man¬ 
ually  from  direct  observations  and  ergonomic  measurements  (see  Appendix  A  for  a  summary). 

The  K-Wall  was  located  in  front  of  the  “command  table”  in  the  JCC  (see  Figure  4),  so  that  its  pri¬ 
mary  users  were  the  players  that  comprised  the  CJTF  staff  (CJTF,  CoS,  J2,  J3,  J4,  J6,  BWC,  BWl, 
BW2).  Other  players  that  were  located  in  the  JCC,  including  those  in  the  cells,  could  view  the  K- 
Wall,  but  this  required  them  to  turn  around  from  their  own  computer  systems — systems  that  were 
considerably  more  distant  to  the  K-Wall.  Therefore,  the  observational  data  collected  came  predomi¬ 
nantly  from  the  primary  K-Wall  users.  However,  observations  of  and  comments  made  by  other  game 
players,  including  the  functional  area  Commanders,  functional  area  anchor  desk  personnel  (i.e.,  the 
information  providers  to  K-Wall),  and  visitors  to  the  JCC  were  also  collected  when  available. 

2.2.1.  Automated  Data  Collection 

Automated  data  were  collected  in  order  to  track  Universal  Resource  Location  (URL)  access  and 
changes  in  the  configuration  of  displays  of  the  K-Wall.  This  was  an  attractive  means  of  collecting 
data  because  it  was  completely  non-invasive.  Two  types  of  automated  data  collection  were  imple¬ 
mented.  Logs  of  the  usage  on  each  display  of  the  K-Wall  were  created  at  the  beginning  of  each  day 
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and  updated  any  time  a  change  was  made.  These  logs  included  the  address  of  the  Web  pages  that 
were  displayed  on  the  K-Wall,  when  the  URLs  to  these  pages  were  accessed,  and  on  which  K-Wall 
monitor  they  were  displayed.  The  usage  logs  were  also  updated  every  time  a  K-Wall  operation  was 
performed,  such  as  when  the  contents  in  one  of  the  peripheral  monitors  was  brought  into  focus  on 
one  of  the  central  monitors,  when  windows  were  manually  refreshed,  minimized,  or  maximized,  and 
when  one  of  four  preset  configurations  of  information  was  chosen^.  The  usage  logs  did  not  include 
information  about  other  tools  or  software  that  was  displayed  on  the  K-Wall,  unless  they  were 
accessed  via  the  K-Wall  application  (i.e.,  via  a  URL). 

The  second  type  of  automatically  collected  data  came  from  screen  captures  of  the  entire  K-Wall, 
which  provided  insights  into  its  usage  by  showing  actual  content  and  configuration.  These  were  also 
time  stamped  and  they  were  taken  automatically  every  5  minutes. 

2.2.2.  Observational  Data  Collection 

As  is  usually  the  case  when  collecting  data  non-invasively  in  a  real-world,  operational  setting, 
much  of  the  data  collection  was  informal  and  observational — particularly  given  the  constraints  dis¬ 
cussed  above.  However,  this  kind  of  data  often  proves  extremely  informative  due  to  its  broader  scope 
and  in-context  nature.  Observational  data  collected  during  the  game  related  to  several  different 
aspects  of  the  K-Wall,  including: 

•  Critical  events  and  situations,  which  allowed  the  other  data  to  be  put  into  a  context  over  time. 
Critical  events  were  broken  down  into  (a)  those  events  that  caused  a  modification  in  the  usage 
pattern  of  the  K-Wall,  and  (b)  those  events  that  were  determined  to  be  mission-critical  at  the 
daily  outbriefs  and  based  on  observer  comments. 

•  K-Wall  usage  patterns,  including  which  tools  and  URLs  K-Wall  users  accessed  and  how  the 
K-Wall  displays  are  configured. 

•  User  preferences,  in  terms  of  how  much  users  liked  or  disliked  the  K-Wall  and  its  features. 

•  K-Wall  usability,  including  issues  related  to  the  ease  and  intuitiveness  of  interacting  with  the 
K-Wall. 

•  Suggestions  for  new  requirements,  in  terms  of  the  content,  features,  and  tools  of  the  K-Wall. 

•  Visibility  issues,  including  the  legibility  of  text  and  graphics  on  the  K-Wall,  etc. 

•  Ergonomic  issues  related  to  the  K-Wall,  such  as  those  associated  with  occlusions,  monitor 
glare,  room  layout,  communication  between  personnel,  etc. 

Each  of  these  data  sets  was  gathered  from  different  sources.  For  example,  issues  related  to  the 
update  rates  of  the  summary  pages  could  be  established  from  a  range  of  observations  including  both 
the  automated  data  collection  (described  above)  and  records  of  overt  verbalizations.  Other  data,  such 
as  user  preferences,  could  only  be  obtained  via  interrogation  and  from  verbal  comments.  The  specific 
data  sources  used  to  collect  observational  data  included: 

1 .  Logging  of  critical  events  that  caused  a  change  in  K-Wall  use. 


^  Hiding  a  K-Wall  window  allowed  for  other  non-Web  based  software  (primarily  C2PC)  to  be  displayed  in  its  place. 
The  presets  were  designed  to  give  users  a  tactical  focus  by  hiding  at  least  one  of  the  focus  windows.  The  four  presets 
were:  (1)  all  windows  shown  except  9  (left  focus  window)  and  10  (right  focus  window)  hidden;  (2)  all  windows 
shown  except  8  (bottom  left  window)  and  9;  (3)  all  windows  shown  except  10  and  1 1  (bottom  right  window);  and 
(4)  all  windows  shown  except  8,  9,  10,  and  1 1  hidden. 


2.  Time  sampling  of  the  content  and  configuration  of  the  K-Wall,  as  well  as  which  tools  and 
summary  pages  were  being  used  or  viewed  (discussed). 

3.  Observations  of  verbal  and  nonverbal  behaviors  of  the  K-Wall  users  as  they  related  to 
usability  issues,  new  requirements,  preferences,  and  visibility. 

4.  Probing  of  specific  K-Wall  users  about  the  above  issues  (usability,  requirement,  preference, 
and  visibility)  if  observations  of  behavior  did  not  provide  enough  information. 

5.  Measurements  and  descriptions  of  the  variables  that  related  to  the  K-Wall’s  ergonomic  use. 

2.3.  SCHEDULE  OF  DATA  COLLECTION 

SSC  San  Diego  observers  were  located  in  the  JCC  itself,  in  close  proximity  to  the  K-Wall  and  its 
primary  users  (see  the  JCC  layout.  Figure  4),  allowing  data  to  be  collected  almost  continuously 
during  the  game.  The  specific  data  collection  techniques  were  employed  according  to  a 
predetermined  schedule: 

1 .  The  logs  of  critical  events  and  situation  changes  were  updated  ft-om  moment  to  moment,  as 
they  occurred. 

2.  Automated  logs  of  URL  access  and  changes  in  the  configuration  of  displays  of  the  K-Wall 
were  updated  any  time  a  change  occurred. 

3.  Screen  captures  of  K-Wall  content  were  automatically  taken  every  5  minutes. 

4.  Time  sampling  of  usage  patterns  (tool  use  and  K-Wall  monitor  content)  were  manually 
recorded  every  hour. 

5.  Observational  data  collection  and  probing  of  user  responses  regarding  usability  issues,  new 
requirements,  preferences,  and  visibility  across  most  primary  K-Wall  users  occurred 
throughout  the  game. 

2.4.  OTHER  SOURCES  OF  OBSERVATIONAL  DATA 

The  reported  data  consists  primarily  of  that  data  collected  using  the  methods  described  above. 
However,  several  other  sources  of  data  that  included  observations  about  the  K-Wall  and  its  usage 
during  the  game  were  also  made  available  (by  game  personnel  and  other  data  collection  teams 
working  at  Global  2000)  and  the  data  from  these  sources  is  also  included.  These  sources  are 
discussed  below. 

Knowledge  Managers 

The  observational  data  include  comments  made  during  discussions  with  the  Knowledge  Managers 
(KMs)  and  made  during  the  daily  KM  meetings.  In  Global  2000,  a  team  of  personnel  was  given  the 
role  of  supporting  and  evaluating  knowledge  management.  One  of  the  many  responsibilities  of  this 
role  included  supporting  players  in  the  effective  use  of  the  tools  made  available  to  them  during  the 
game.  The  KMs  provided  invaluable  insights  into  many  issues  related  to  the  K-Wall  because  of  their 
roles  as  facilitators  for  the  use  of  the  K-Wall  and  other  information  technologies.  These  perceptions 
were  also  documented  and  made  available  to  members  of  the  Command  21  program  team  during 
Global  2000.  The  documented  comments  also  included  a  survey  that  contained  a  question  specifi¬ 
cally  asking  about  the  K-Wall  and  its  impact.  These  data  were  extremely  useful  because  they  pro¬ 
vided  insight  into  the  perceptions  and  issues  expressed  by  the  players  not  located  in  the  JCC,  includ¬ 
ing  those  of  the  information  providers  that  produced  the  summary  pages  that  were  displayed  on  the 
K-Wall. 
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WIGS 


Documents  stored  on  WIGS  were  also  made  available  and  provided  important  information.  This 
was  a  Web-based  system  that  allowed  players  to  access  various  game-related  documents,  such  as 
daily  briefs,  rules  of  engagement,  news  updates,  and  functional  area  summary  pages.  Game  players 
could  access  any  of  the  documents  located  in  WIGS  using  their  Web  browser.  The  most  valuable 
information  available  on  WIGS,  in  terms  of  the  current  data  collection  effort,  was  the  situation  sum¬ 
maries  (the  SitSum)  that  were  updated  continuously.  This  information  allowed  critical  events  to  be 
identified,  which  enabled  the  manual  recording  of  these  events  to  be  related  in  time  to  both  the  auto¬ 
mated  and  observational  data  collection. 

Executive  Sessions 

Discussions  during  preparation  for  post-game  Executive  Sessions  and  during  the  sessions  them¬ 
selves  included  many  important  comments,  from  primary  K-Wall  users  and  others,  regarding  the  K- 
Wall  and  its  impact  during  the  game. 
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3.  RESULTS 


Appendix  A  contains  all  of  the  observational  data  collected  during  Global  2000.  These  comments 
fell  into  four  main  categories,  which  related  to: 

1 .  The  initial  requirements  identified  in  previous  interviews  and  which  the  K-Wall  was  designed 
to  meet. 

2.  The  design,  content,  and  tools  of  the  summary  pages. 

3.  Usability  and  usage  of  the  K-Wall. 

4.  Other  emerging  themes. 

The  results  are  organized  around  these  themes  and  include  discussion  of  the  observational  data,  as 
well  as  any  appropriate  analyses  conducted  on  the  data  collected  automatically  (i.e.,  the  K-Wall 
usage  logs  and  screen  captures).  Appendix  B  contains  tables  of  the  automated  data  presented  in  the 
figures  in  the  results  section. 

3.1.  INFORMATION  REQUIREMENTS 

As  discussed  above,  a  series  of  structured  interviews  with  JOC  senior  staff  conducted  by  SSC  San 
Diego  (Moore  &  Averett,  1999;  Smallman  et  al.,  2001)  led  to  the  identification  of  14  critical  user 
requirements  that  the  K-Wall  must  meet.  These  requirements,  given  in  Table  1,  were  grouped  into 
four  categories:  General,  Format,  Content,  and  Features.  One  goal  of  the  current  data  collection  effort 
was  to  determine  how  well  the  initial  K-Wall  implementation  met  these  requirements.  Specifically,  it 
attempted  to  determine  whether: 

•  The  K-Wall  design  solutions  support  the  user  requirements  (see  Table  1). 

•  The  interviews  identified  the  right  information  requirements. 

•  There  are  additional  requirements  that  will  require  new  K-Wall  design  features  and  content. 

Each  of  the  14  user  requirements  identified  in  the  interviews  is  discussed  below,  with  para¬ 
phrased  representative  comments  and  supporting  data,  when  available,  that  indicate  how  well 
each  requirement  was  met. 

3.1.1.  General  Requirement  1:  Situation  Awareness 

The  K-Wall  attempted  to  meet  users’  need  for  shared  SA  by 
bringing  multiple  sources  of  information  together  on  a  single 
shared  display.  Unfortunately,  no  formal  measures  of  SA  could  be 
taken  during  Global  2000.  This  was  due  to  constraints  imposed  on 
the  data  collection  procedures  used — questionnaires  given  to  the 
players  or  other  formal  procedures  would  have  disrupted  game 
play.  However,  users  perceptions  of  how  well  SA  was  supported 
can  be  discerned  from  the  observed  comments  of  JCC  staff.  All  of 
these  comments  should  be  considered  in  light  of  the  fact  that  there 
was  a  limited  number  of  tools  available  to  the  game  players  which 
they  could  take  advantage  of  to  acquire  and  maintain  SA.  In 
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general,  most  of  the  relevant  comments  made  by  K-Wall  users  suggested  that  they  believed  that  the 
K-Wall  was  the  best  tool  available  to  them  for  the  acquisition  of  SA  and  that  it  provided  support  for 
SA  reasonably  well.  In  particular,  the  comments  reflect  the  perception  that  personnel  who  had  access 
to  the  K-Wall  were  able  to  acquire  SA  more  easily  than  those  who  did  not  have  this  access.  Typical 
comments  included: 

•  It  was  useful  for  the  CJTF;  it  was  the  only  tool  available  . . .  normally  he  would  get  SA  from 
radios,  people  talking  to  each  other  . . .  here  the  K-Wall  was  the  only  way  to  get  SA.^ 

•  We  need  a  common  picture  that  gives  instant  SA  to  the  commander  when  they  walk  into  the 
room  . .  .  this  is  a  pretty  interesting  first  step. 

•  Life  would  have  been  miserable  if  I  didn’t  have  this  . . .  because  I  would  have  been  constantly 
pinging  the  other  guys  about  what’s  going  on. 

•  The  CJTF  did  not  have  to  2nd  guess  commanders  because  of  comfort  he  had. . . . 

•  SA  increased  in  the  CJTF  but  not  component  commands. 

•  We  didn’t  do  a  briefing  at  8  o’clock,  1  o’clock  ...  we  didn’t  need  to  be  brought  up  to  date 
because  we  were  always  up  to  date. . . . 

•  Speed  of  Command  was  increased  by  the  improved  SA  of  the  CJTF.  The  K-Wall  was  a  factor 
in  terms  of  improved  speed  of  command.  SA  was  not  good  in  the  functional  cells. 

•  [Regarding  “working  information  into  knowledge’’]  When  I  walked  around  during  game  play, 
this  [the  K-Wall]  was  the  only  place  it  was  happening. 

•  The  K-Wall  is  valuable  in  giving  all  a  sense  of  what’s  going  on,  especially  the  CJTF  and  the 
CINC. 

•  The  K-Wall  does  a  reasonable  job  of  keeping  awareness  level  to  the  CJTF  and  his  staff. 

•  I  found  the  K-Wall  useful.  The  CJTF  depends  on  the  K-Wall  almost  entirely  for  awareness - 

3.1.2.  General  Requirement  2:  integrated  Information 

Another  general  requirement  that  the  K-Wall  was  designed  to 
meet  was  the  need  to  display  integrated  information.  Again,  the 
design  of  the  K-Wall  addressed  this  need  by  bringing  multiple 
sources  of  information  together  on  a  single  display.  The  observed 
comments  made  by  K-Wall  users  suggest  that  they  believed  that 
displaying  information  typically  distributed  across  multiple  systems 
simultaneously,  in  a  single  display,  was  a  step  toward  providing 
integration  of  the  information  provided  by  the  different  functional 
areas.  However,  the  comments  also  reflect  the  feeling  that  it  was 
still  difficult  to  fuse  the  information  that  was  displayed  on  the  K- 
Wall  to  form  a  sufficiently  integrated  picture  of  the  operational 
situation  across  functional  areas.  Comments  related  to  this  issue 
included: 

•  Only  the  CJTF  can  look  at  all  the  screens  at  the  same  time.  [Those  without  a  K-Wall 
cannot. . . .] 

•  I  see  this  as  valuable  because  we  can  put  different  formats  up  there  to  discuss  things. 
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•  The  K-Wall  was  a  good  first  step.  We  need  better  integration  ...  it  was  hard  to  see  every¬ 
thing.  . . . 

•  The  CJTF  is  the  integrator  because  of  more  screens. 

•  An  advantage  we  had  over  the  component  commanders  was  not  having  a  single  page  but  multi¬ 
ple  screens. 

•  It’s  hard  to  integrate  the  stoplights  [status  alerts]. . . . 

•  It  allowed  us  to  look  at  the  same  information  at  the  same  time  and  discuss  it. 

•  The  information  presented  on  the  wall  was  not  fused  from  relevant  functional  areas.  However, 
it  gave  the  false  impression  that  it  was. . . . 

3.1.3.  Format  Requirements  1  and  2:  Intuitive,  Graphical,  and  Consistent 

The  interviews  revealed  a  need  for  the  information  presented  on 
the  K-Wall  to  be  in  a  consistent  format  that  was  intuitive  and 
graphical.  In  order  to  meet  this  need,  information  providers  were 
trained  to  populate  the  K-Wall  with  summary  pages,  html  pages  that 
had  a  predetermined  layout  and  format.  A  later  section  of  the 
Results  is  devoted  to  issues  related  to  the  content  and  format  of 
these  summary  pages  (see  section  3.2.2).  In  order  to  support 
graphical  information  on  the  K-Wall,  the  functional  area  anchor 
desk  personnel  were  encouraged  to  use  graphical  representations  of 
information  when  possible.  They  were  also  provided  with  the 
TacGraph  tool  with  which  they  could  provide  graphical  tactical 
information  to  be  published  on  the  K-Wall.  The  usage  of  this  tool  is 
discussed  in  sections  3.1.4  and  3.1.13.  The  comments  regarding  the  format  of  the  K-Wall  suggest 
that,  although  the  format  of  the  K-Wall  was  “fairly”  intuitive,  there  were  problems  with  consistency 
that  were  not  resolved  during  the  game.  These  problems  stemmed  from  a  lack  of  standardization  of 
the  representation  of  information  on  the  K-Wall,  especially  across  the  different  military  forces. 
Representative  comments  included: 

•  The  technology,  a  lot  of  it  being  the  K-Wall,  was  fairly  intuitive.  .  .  . 

•  An  observed  problem  is  that  navy  personnel  don’t  know  what  the  army  symbols  are — a 
suggestion  was  made  that  there  should  be  a  roll-over  function  in  TacGraph  that  shows  a 
description  of  a  unit. 

•  There  is  no  way  to  have  one  metric  for  all  the  functional  areas. ...  If  we  were  together  for  a 
longer  time,  a  metric  would  develop. 

•  We  need  standards  of  use  (e.g.,  colors  used,  processes  completed,  level  of  detail,  purposes, 
etc.). . .  . 

•  We  need  to  determine  what  level  of  information  you  see  at  first  blush. 
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3.1.4.  Content  Requirement  1:  Tactical  Focus 

A  desire  expressed  by  users  in  the  interviews  was  for  multiple 
tactical  displays  as  the  focus  of  the  K-Wall.  The  K-Wall  provided 
two  large,  side-by-side  focus  monitors  that  were  intended  to  meet 
the  need  for  a  tactical  focus.  The  primary  software  used  to  generate 
detailed  tactical  information  during  Global  2000  was  C2PC. 
Another  common  way  that  information  producers  represented 
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tactical  information,  at  a  less  detailed  level  than  C2PC,  was  to  link  pictures  produced  using  the 
TacGraph  tool  to  the  functional  area  summary  pages. 

Tactical  information  was  included  in  the  focus  of  the  K-Wall  for  a  significant  proportion  of  the 
game.  Analysis  of  the  screen  captures  of  the  K-Wall  revealed  that  tactical  information  provided  by 
C2PC  or  TacGraph  was  displayed  on  at  least  one  of  the  K-Wall  focus  monitors  34%  of  the  time. 
However,  tactical  data  was  rarely  the  only  focus  of  the  K-Wall— fcot/i  focus  monitors  contained 
tactical  information  only  8%  of  the  time.  Figure  8  shows  the  percent  of  time  that  C2PC  was  running 
or  a  summary  page  link  containing  a  TacGraph  picture  was  being  displayed  on  one  or  both  of  the 
focus  monitors  as  a  function  of  the  operational  phase  of  the  game. 


Tactical  Focus 


C2PC  in  both  C2PC  in  one  TacGraph  in  TacGraph  in 
Focus  Focus  both  Focus  one  Focus 

windows  window  windows  window 


Figure  8.  The  percentage  of  time  that  tactical  information  was  dis¬ 
played  in  one  or  both  of  the  K-Wall  focus  displays  (Monitors  9  and  10) 
as  a  function  of  format  of  information  and  operational  phase. 


3.1.5.  Content  Requirement  2:  Supplemental  Information 

In  order  to  display  other  information  besides  tactical,  the  K-Wall  design  provided  10  small, 
peripheral  monitors  surrounding  the  two  focus  monitors  (see  Figure  1).  The  observed  behavior  of  the 
K-Wall  users  demonstrated  both  confirmation  of  the  need  for  this 
information  and  a  potential  design  problem  with  K-Wall  layout. 

Although  the  information  on  these  monitors  was  frequently  viewed 
or  discussed,  it  was  almost  always  brought  into  focus  on  one  of  the 
larger  central  monitors  in  order  for  K-Wall  users  to  be  able  to  read 
it.  This  was  especially  the  case  when  any  links  from  the  summary 
pages  were  being  viewed.  This  behavior  suggests,  that  although 
users  were  very  interested  in  seeing  non-tactical  information  on  the 
K-Wall,  the  small  peripheral  monitors  only  provided  some 
information  (such  as  changes  in  the  color  of  the  status  indicators  or 
in  the  color  of  the  links  text),  and  that  they  were  not  sufficiently 
large  to  provide  the  information  that  users  wanted  to  see  and  read 
(also  see  the  discussion  of  visibility  issues  in  section  3.3.2). 
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3.1.6.  Content  Requirement  3:  Mission  Goals  and  Objectives 

A  requirement  identified  in  the  interviews  was  for  the  K-Wall  to 
provide  access  to  mission  goals  and  objectives.  A  goal  of  the  K- 
Wall  design  was  to  meet  the  need  to  visualize  goals,  plans,  and 
mission  status,  however  no  single  monitor  or  location  was  dedicated 
to  provide  this  information.  The  information  providers  did  make 
this  type  of  information  available  in  links  from  the  functional  area 
summary  pages  and  there  were  also  separate  summary  pages  for  the 
JFLEX  and  TAPS  tools,  which  provide  mission  planning  and 
effects  evaluation.  Further,  the  CJTF  staff  developed  their  own 
summary  page  that  was  intended  to  provide  this  sort  of  information 
to  both  the  CINC/NCA  and  functional  area  component 
commanders.  Comments  made  by  the  K-Wall  users,  however, 
reflect  a  perception  that  the  K-Wall  did  not  sufficiently  support  communication  of  mission  goals  and 
objectives.  These  comments  include: 

•  It  did  not  provide  the  commander’s  intent  to  me. 

•  The  CJTF  level  did  not  feel  comfortable  .  .  .  with  no  way  to  sync  with  higher  and  lower 
levels — no  way  to  push  information  or  convey  needs/plans/intentions. 

•  When  the  component  commanders  let  the  CJTF  know  they  were  having  trouble  knowing  what 
the  CJTF  wanted,  it  led  to  the  development  of  a  CJTF  page. 

•  Need  to  define  better  what  “Today,”  ‘Tomorrow,”  and  “Long  Range”  mean.  Some  areas  (e.g.. 
Sea  Control)  used  them  better  than  others.  Information  requirements  and  CCIRs  should  be 
considered. 

•  Commander’s  priorities  must  be  made  clear  to  all.  Shared  awareness  must  include 
CCIRS/priorities  etc.,  or  it’s  not  shared. 

•  The  overall  battle  plan  is  not  really  reflected  on  the  K-Wall. . . . 

3.1.7.  Content  Requirement  4:  Anchor  Desk  Output 

Previous  interviews  indicated  that  K-Wall  users  needed  to  see 
output  from  anchor  desks  on  the  supplemental  monitors  of  the  K- 
Wall.  Analysis  of  the  screen  captures  revealed  that  information 
produced  by  the  anchor  desks  from  the  13  functional  areas  was  the 
primary  information  presented  on  the  peripheral  monitors  of  the  K- 
Wall.  Figure  9  (top  panel)  shows  the  percentage  of  time  that  a 
summary  page  was  displayed  on  the  K-Wall  displays.  Even  though 
many  types  information  and  tools  were  available  to  be  displayed  on 
the  K-Wall,  Figure  9  reveals  that  the  summary  pages  were  by  far 
the  most  commonly  displayed  information  products  (86%,  averaged 
across  all  monitors).  The  bottom  panel  shows  the  percentage  of  time 
that  the  summary  pages  from  the  individual  functional  areas  were 

displayed  on  the  K-Wall  displays,  averaged  across  monitors.  As  can  be  seen,  all  of  the  functional 
area  summary  pages  were  displayed  on  the  K-Wall,  although  the  amount  of  time  that  they  were 
displayed  varied  across  the  different  areas. 
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Figure  9,  Top  panel  shows  percentage  of  time  that  a  summary  page 
was  displayed  on  the  K-Wall  displays,  averaged  across  all  monitors. 
Bottom  panel  shows  percentage  of  time  that  a  summary  page  from 
one  of  the  13  functional  areas  was  displayed  on  the  K-Wall, 
averaged  across  all  monitors. 


The  K-Wall  data  logs  also  indicated  that  information  products  from  these  anchor  desks  were  the 
most  commonly  accessed  information  on  the  K-Wall  monitors.  Figure  10  shows  the  number  (per 
hour)  of  summary  pages  and  other  information  products  (i.e.,  links  from  the  summary  pages) 
produced  by  the  functional  areas  that  were  accessed  as  a  function  of  operational  phase.  As  can  be 
seen,  over  time,  as  the  K-Wall  users  and  information  providers  became  more  familiar  with  the 
technology,  they  accessed  more  K-Wall  information  products. 
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Figure  10.  The  number  of  summary  pages  and  other  informa¬ 
tion  products  (i.e.,  links)  from  the  13  functional  areas  that  were 
accessed  via  the  K-Wall,  as  a  function  of  operational  phase. 

3.1.8.  Content  Requirement  5:  Connectivity/Collaboration 

During  Global  2000,  several  collaboration  tools  were  made  avail¬ 
able  to  players.  The  primeiry  tool  provided  was  InfoWorkSpace 
(IWS),  a  bundle  of  collaborative  tools  for  communication  and  data 
access.  K-Wall  users  who  had  access  to  IWS,  however,  tended  to 
use  their  IWS  on  the  laptops  provided  on  the  command  table.  The 
K-Wall  did  provide  a  dedicated  VTC  window  (on  a  thirteenth 
monitor)  but  this  was  not  used  for  this  purpose.  Comments  that 
related  to  collaboration  and  connectivity  during  the  game  in  general 
were: 

•  Groups  that  were  on  IWS  did  use  chat  for  collaboration. 

•  There  was  a  need  for  better  shared  awareness/collaboration. 

•  The  VTC  was  worthless,  actually  negative  because  poor  execution  made  it  a  distraction  . . . 
maybe  they  thought  they  were  sharing  information  but  they  weren’t.  It  should  only  be  used  in 
certain  situations. 

•  What  made  it  work  was  the  table,  people  around  the  table,  verbal  communication  . . .  great  for 
passing  information  up  and  down  . . .  but  I  missed  face  to  face  and  verbal  communication  from 
commanders  to  work  . . .  needed  to  get  the  headphones  to  work,  the  VTC  to  work. 
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3.1.9.  Content  Requirement  6:  Cognitive  Support 

The  K-Wall  itself  did  not  provide  integrated  cognitive  support 
tools.  Therefore,  most  comments  related  to  cognitive  tools  focused 
around  the  need  for  such  tools  on  the  K-Wall  in  general  or  on  the 
summary  pages  themselves.  These  requirements  for  cognitive 
support  are  discussed  in  later  sections  (New  K-Wall  Tools  and 
Summary  Page  tools). 

The  output  (in  the  form  of  summary  pages  created  for  this  purpose 
during  the  game)  from  a  limited  number  of  noninte grated  cognitive 
support  tools  could  be  displayed  on  the  K-Wall.  These  tools  were 
the: 

•  Theater  Assessment  Profiling  System  (TAPS),  which  pro¬ 
vided  effects-based  assessment  of  the  operational  system. 

•  Joint  Force  Level  Execution  Monitoring  and  Re-planning  Software  (JFLEX),  a  mission 
planning  and  monitoring  tool. 

•  Computer  Aided  Evaluation  of  System  Architectures  (CAESAR),  a  course  of  action  visualiza¬ 
tion  support  tool  (Levis,  2000). 

When  they  were  accessed,  these  tools  or  their  outputs  were  most  often  displayed  on  one  of  the 
focus  monitors  of  the  K-Wall.  Figure  1 1  shows  the  percentage  of  time  that  the  output  from  one  of 
these  cognitive  support  tools  was  displayed  on  one  of  the  K-Wall  focus  monitors. 
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Figure  1 1 .  The  percent  of  time  that  output  of  the  few  cognitive  tools  was 
displayed  on  one  of  the  K-Wall  focus  monitors. 

As  can  be  seen,  K-Wall  users  rarely  viewed  the  output  of  these  tools — the  most  often  accessed  tool 
was  TAPS,  which  was  accessed  only  3.6%  of  the  time  during  the  game  (averaged  across  all  three 
phases).  Because  output  from  the  other  tools  (JFLEX  and  CAESAR)  was  so  infrequently  displayed 


20 


on  the  K-Wall,  no  user  comments  were  observed  that  related  to  these  tools.  Comments  related  to  the 
TAPS  tools  included: 

•  TAPS  would  be  nice,  if  it’s  working. ...  Is  TAPS  updated  on  here? 

•  A  common  operational  baseline  should  exist,  a  shared  story. .  .  it  could  be  TAPS  at  the  NCA 
level. 

•  Great  idea  but  information  time-late  and  manpower  intensive.  It  should  be  automated.  Imma¬ 
ture  tool — useless  now  but  maybe  good  some  day. 

•  I  don’t  know  what  this  is  telling  me! 

•  This  [TAPS]  is  a  bean  counter’s  tool. 

•  I  don’t  trust  it  because  I  don’t  understand  it. 

•  As  executed  in  game,  this  is  a  very  immature  tool.  Looks  like  it  should  have  been  good,  but 
information  inputted  was  from  different  people  with  different  subjective  opinions.  If  you  don’t 
know  where  you  are  going,  any  road  will  get  you  there.  . .  . 

•  No  decision  points,  no  triggers,  no  sense  of  progress — maybe  it  should  be  a  decision  aid.  This 
is  way  ahead  if  its  time. 

Suggestions  for  several  new  K-Wall  tools  were  made  throughout  the  course  of  the  game.  Tools  for 
the  K-Wall  in  general  that  were  requested  were  those  that  provided: 

•  Alerting — tools  that  allowed  the  K-Wall  users  to  alert  others  about  information  and  to  be 
alerted  about  changes,  especially  in  information  not  displayed  on  the  K-Wall,  were  requested. 

•  Attention  management  (pointers  etc.) — many  requests  were  made  to  provide  tools  to  direct  K- 
Wall  users’  attention  to  relevant  information,  especially  in  the  context  of 
discussion/collaboration  using  the  K-Wall. 

•  Other  tools — included  chat  capability  with  information  providers  (e.g.,  Webcam32). 

Related  comments  included: 

•  There  have  been  lots  of  requests  for  the  CJTF  to  have  control  of  alerts — to  be  able  to  alert 
everyone  else  (i.e.,  push  information  to  everyone  else). 

•  The  pages  that  aren’t  displayed  on  the  K-Wall  don’t  get  looked  at. 

•  [To  collaborate. . .  ]  We  each  need  to  have  our  own  different  colored  pointers  so  we  can  talk 
about  the  same  things  together. 

•  You  need  the  ability  to  have  a  pen  or  light  you  can  use  to  draw  on  board  from  a  distance  ...  a 
mouse  pointer/pencil  device. 

•  Built-in  chat  with  functional  areas  for  summary  pages  would  be  useful. 

•  Webcam32  might  be  a  tool  for  future  use  on  the  wall. 

Content  Requirements.  The  content  of  the  K-Wall  was  identified  as  the  key  to  its  successful  use  by 
its  users  as  evidenced  by  general  comments  that  were  overheard  such  as: 

•  Content  and  people  make  the  K-Wall  useful — without  them,  it  is  simply  a  bunch  of  monitors. 

•  Garbage  in-garbage  out. 

As  before,  the  K-Wall  was  designed  to  meet  the  content  requirements  that  emerged  from  previous 
interviews  with  potential  K-Wall  users. 
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3.1.10.  Feature  Requirement  1 :  Flexible  Configuration 

The  K-Wall  provided  flexibility  in  the  configuration  of  the 
displays  by  allowing  users  to  easily  move  the  contents  of  the  K- 
Wall  monitors  and  bring  up  new  information  as  necessary.  The  ini¬ 
tial  intent  of  the  K-Wall  design  was  that  some  monitors  would  be 
dedicated  to  presenting  the  same  type  of  information  and  that  others 
would  be  configurable,  and  could  be  changed  as  the  context 
demanded.  However,  in  practice,  all  the  monitors  were  configurable 
and  could  display  any  information  available  to  K-Wall  users. 

The  comments  and  behaviors  of  the  K-Wall  users  reflected  two 
contexts  within  which  the  need  for  configurable  displays  must  be 
met.  First,  the  K-Wall  must  be  configurable  to  support  a  change  in 
the  operational  situation.  During  Global  2000,  K-Wall  users  were  observed  to  reconfigure  the  K-Wall 
displays  to  meet  this  need.  For  example,  they  brought  up  different  summary  pages  in  the  peripheral 
monitors  following  a  change  in  operational  phase. 

Second,  the  configuration  of  the  K-Wall  must  be  flexible  enough  to  meet  the  demands  of  different 
users.  Much  discussion  revolved  around  this  requirement  for  user-defined  K-Wall  configuration  (see 
the  discussion  of  Tailorable  K-Walls  in  section  3.4.1)  However,  the  fact  that  the  K-Wall  was  used  by 
only  two  groups  (composed  of  the  same  individuals)  during  Global  2000  made  it  difficult  to  collect 
data  relating  to  this  requirement.  At  the  NWC,  the  K-Wall  was  configured  to  meet  the  information 
requirements  of  the  CJTF  and  JCC  staff.  However,  the  CINC  could  have  configured  the  duplicate  K- 
Wall  aboard  the  Coronado  differently  for  use.  A  comparison  of  the  typical  configurations  of  these 
two  walls  revealed  that  they  were  configured  quite  similarly  (see  discussion  on  K-Wall  Usage  on 
Coronado,  section  3.4.3) — though  the  reasons  for  this  are  unclear. 

Across  Situations 

•  Now  that  we  are  no  longer  in  [Phase  1]  and  in  [Phase  2],  I  need  to  see  information  relevant  to 
that .  . .  only  certain  pages. 

•  We  need  to  replace  that  Overview  with  ground  control 

•  At  the  beginning  of  Phase  2,  the  Overview  page  was  no  longer  being  displayed,  and  no  one 
seemed  to  use  it .  .  .  later  in  Phase3,  the  Overview  page  was  brought  back  up. 

•  The  limitations  were  lack  of  flexibility  in  reconfiguring  it.  .  .  . 

Across  Users 

•  Whose  tool  is  it?  If  three  groups,  CJTF,  CENC,  NCA,  can  see  the  K-Wall  they  need  to  see 
different  information. .  .  .  One  K-Wall  glove  does  not  fit  all — there  needs  to  be  tailorable 
information.  .  .  . 

•  The  big  issue  is  who  uses  it.  Information  presented  should  differ.  It  should  be  tailored  to  spe¬ 
cific  users/user  groups. 
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3.1.11.  Feature  Requirement  2:  Drill-Down 

The  Web-based  K-Wall  application  and  the  summary  pages  pro¬ 
duced  by  the  functional  area  information  providers  were  designed  to 
support  drill-down  to  more  detailed  information.  The  intent  of  this 
design  was  to  enable  information  providers  to  use  the  bulleted  text 
of  the  summary  pages  (status  alerts,  alerts,  impacts,  links)  as  hyper¬ 
links  to  other  information  products  such  as  other  Web  pages,  briefs, 
and  other  documents.  As  the  game  progressed  and  information  pro¬ 
viders  became  more  adept  at  posting  to  the  K-Wall  and  updating 
summary  pages,  the  number  of  the  links  increased,  content  of  these 
links  became  more  detailed,  and  the  relationship  between  linked 
products  became  more  complex.  This  is  reflected  in  the  K-Wall  data 
logs,  which  automatically  recorded  the  path  of  every  link  accessed 
through  the  K-Wall  application,  from  which  can  be  extracted  both  the  source  and  format  of  these 
links.  Figure  12  shows  the  content  and  format  of  the  pages  accessed  as  functions  of  the  operational 
phase  of  the  game.  As  can  be  seen,  the  majority  of  these  pages  were  Web  pages,  in  html  format. 
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Figure  12.  The  number  of  links  accessed  per  hour  of  game  play  via  the  K- 
Wall  as  a  function  of  format  and  operational  phase. 

Figure  13  shows  the  number  of  products  (html  pages,  PowerPoint  briefs.  Word  documents,  etc.) 
that  were  produced  by  functional  area  information  providers  as  a  function  of  the  operational  phase. 
The  top  panel  shows  the  number  of  summary  pages  that  were  accessed  per  hour.  The  lower  panel 
shows  the  number  and  kinds  of  other  products  produced  by  the  functional  areas  anchor  desks  (i.e., 
those  that  were  not  summary  pages).  These  pages  were  those  that  were  accessed  via  drill-down  from 
the  summary  pages.  As  can  be  seen  from  these  figures,  a  great  deal  of  drill-down  occurred — over  700 
links  were  accessed  over  the  course  of  the  game  (=40  hours  of  total  game  time).  Although,  as  would 
be  expected,  the  relative  amount  that  information  products  from  the  different  functional  areas  were 
accessed  varied  as  a  function  of  phase,  the  number  of  surmnary  pages  and  links  accessed  across  all 
the  functional  areas  increased  over  time. 
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Figure  13.  The  number  of  functional  area  summary  pages  and  information 
products  accessed  via  the  K-Wall  as  a  function  of  operational  phase  of  the 
game.  (In  addition  to  the  information  products  from  the  13  functional  areas, 
data  are  shown  for  other  summary  pages:  the  Overview  page  (OVV),  TAPS, 

Blue  Comms  (BC),  CAESAR,  CJTF). 

Although  a  great  deal  of  drill-down  to  access  links  from  summary  pages  was  observed,  many  of 
the  comments  made  by  K-Wall  users  reflected  a  need  for  improvement  in  the  drill-down.  Comments 
reflected  concern  that  drill-down  was  not  being  supported  or  that  the  need  to  drill-down  through  too 
many  levels  to  find  information  might  prove  inefficient.  An  important  observation  made  by  many  of 
the  K-Wall  users  was  that  source  of  infbnnation  was  not  made  apparent  on  the  links.  This  is  an 
important  issue  because  it  suggests  that  users  have  difficulty  maintaining  a  mental  representation  or 
map  of  the  “K-Wall  infonnation  space”  making  it  difficult  to  navigate  through  the  different  levels  of 
linked  information.  Many  comments  were  also  made  regarding  what  the  content  and  format  of  linked 
information  should  be.  More  discussion  related  to  this  topic  can  be  found  in  the  Summary  Page  Links 
Content  and  Format  sections  below  (sections  3.2.1  and  3.2.2).  Representative  comments  included: 

•  There  is  a  problem  with  too  many  links,  1  don’t  want  to  drill-down  to  say  “mother  may  I”. 

•  When  I  click  on  a  red  gumball  there  are  no  links  .  .  .  drill-down  is  not  being  supported. 
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•  [In  a  comment  about  how  intuitive  technology  was]  We  could  view  the  functional  area 
information,  could  drill  down  for  more  information. 

•  The  problem  is  that  pages/links  don’t  indicate  where  they  came  from  . .  .  need  to  include 
addresses  or  headings  on  links. 

•  Left  side  [information  in  focus  monitor  9]  is  from  .  .  .? 


3.1.12.  Feature  Requirement  3:  Information  Age  and  Reliability 

The  K-Wall  provided  information  age,  in  terms  of  the  last  update 
time  of  the  summary  page,  using  a  simple  time  and  date  stamp. 

Early  in  the  game,  problems  with  the  usefulness  of  this  time  stamp 
arose  because  the  computer  systems  (from  which  the  time  stamps 
were  generated)  used  by  players  were  not  all  synchronized  to  show 
the  same  time.  Some  comments  reflected  that  K-Wall  users  wanted 
a  better  indication  of  the  age  of  information  products  presented  on 
the  K-Wall.  More  important,  however,  were  comments  suggesting 
that  users  were  not  happy  with  the  rate  at  which  the  K-Wall  was 
being  displayed,  an  issue  that  will  be  discussed  later  (Summary 
Page  Update),  which  is  related  to  the  age  of  the  information  on  the 
K-Wall.  Representative  comments  included: 

•  One  thing  we’ve  heard  about  the  K-Wall  this  week  is  “how  current  is  the  information?” 

•  [Pointing  out  how  long  it  takes  to  update  page,  and  how  this  leads  to  outdated  information  to 
be  displayed  on  the  K-Wall]  The  problem  is  that  it’s  not  plugged  into  anything  ...  it  takes  so 
long  to  update,  throw  on  overlays,  that  it’s  old  by  the  time  it  gets  up  there. 

•  A  lot  of  these  things  don’t  have  times  on  them  so  we  don’t  know  how  old  they  are. 

•  It’s  been  good  when  the  guys  are  updating  it. 

•  Currency  of  information  is  a  big  issue  with  the  wall,  in  terms  of: 

-  how  long  to  produce  products 

-  how  to  present/link/publish  products 
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3.1.13.  Feature  Requirement  4:  Tactical  Overlays 

The  main  tactical  tool  used  on  the  K-Wall,  C2PC,  provided  a 
number  of  overlays  that  were  used  frequently  throughout  the  game. 

Further,  users  of  TacGraph  could  show  filtered  (de-cluttered) 
representations  and  include  objects  such  as  arrows,  shapes,  etc.,  to 
graphically  represent  things  such  as  projected  movement,  zones  of 
interest,  etc.  Comments  related  to  the  TacGraph  tool  were  made  by 
both  users  of  the  K-Wall  and  the  information  providers.  These 
comments  suggested  that  both  of  these  groups  believed  it  to  be  a 
good  tool  for  creating  useful  graphics  to  represent  the  operational 
picture.  Information  providers  found  the  software  easy  to  interact 
with  to  rapidly  build  effective  graphics.  The  output  of  TacGraph  is 
in  html  format  and  information  providers  took  advantage  of  this, 
sometimes  producing  interactive  graphics  hyperlinked  to  multiple  levels  of  drill-down.  K-Wall  users 
expressed  the  desire  to  see  the  TacGraph  pictures  linked  to  C2PC  data.  Representative  comments 
related  to  C2PC  and  TacGraph  included: 
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•  Clearly  tools  like  TacGraph  are  great  for  rapidly  building  a  picture  that  can  express  info  con¬ 
cisely  and  rapidly. . . . 

•  I  believe  this  is  a  fantastic  tool,  which  was  brought  too  late  in  the  game.  Its  graphics  are  out¬ 
standing  and  bandwidth  is  conserved  when  docs  published  in  JPEG  format. 

•  It  [TacGraph]  needs  to  be  integrated  with  C2PC. 

•  They’re  not  using  TacGraph  enough,  they  should  because  there’s  not  a  lot  of  space  on  page  . . . 
a  picture  is  better  than  words. 

•  SC  summary  graphic  uses  C2PC  backdrop.  What  about  using  TacGraph  or  IMAT,  since  it  can 
show  more  useful  info. 

•  Players  are  actually  trying  to  find  the  full  capability  of  the  systems.  They  are  trying  to  import 
data  from  C2PC  onto  TacGraph.  Both  systems  have  good  features  that  the  other  doesn’t.  They 
want  real  time  data  from  C2PC  on  the  TacGraph  chart  for  manipulation. 

•  Those  procedures  for  providing  the  “picture”  need  refining  . . .  need  training,  not  on  how  to 
operate  tool  but  how  information  can  be  presented  . . .  visualizing  information. . . . 

•  C2PC  isn’t  Web-based — which  is  a  problem  since  everything  else  is. 

•  Bring  up  C2PC  for  situational  awareness. 

3.2.  SUMMARY  PAGES 

The  present  section  of  the  results  is  devoted  to  the  analysis  of  the  observational  and  automated  data 
specifically  related  to  the  summary  pages.  A  great  deal  of  discussion  focused  around  the  content, 
layout,  and  format  of  the  summary  pages.  This  is  not  surprising  since  the  summary  pages  were  the 
most  common  format  of  the  information  displayed  on  the  K-Wall  and  they  served  as  entry  points  into 
the  Knowledge  Web  of  anchor  desk  information  products.  The  current  data  collection  effort 
attempted  to  evaluate  the  design  of  the  summary  pages  and  answer  questions  such  as: 

•  Did  the  summary  pages  (and  their  links)  have  the  right  content? 

•  Did  the  summary  pages  (and  their  links)  have  the  right  format? 

•  How  can  the  summary  pages  be  improved? 

3.2.1.  Summary  Page  and  Links  Content 

A  great  deal  of  discussion  was  focused  around  the  content 
and  features  of  the  summary  pages.  The  summary  pages  of 
two  functional  areas  (ISR  and  Effects)  changed  the  text  of 
the  alerts  (“Today,”  “Tomorrow,”  “Long  Range”),  which  had 
been  intended  to  remain  constant  across  summary  page,  to 
different  text.  However,  most  of  the  discussion  related  to 
how  to  define  specific  content,  who  should  define  it,  and 
how  to  maintain  consistency  across  summary  pages.  This 
suggests  that,  in  general,  the  content  categories  (status,  alerts, 
links,  etc.)  met  the  information  requirements  of  the  K-Wall 
users.  The  discussion  of  the  content  of  the  links  pages  suggested,  not  surprisingly,  that  the  links 
should  provide  information  directly  related  to  the  information  on  the  summary  page.  For  example,  a 
link  from  the  alert  text  for  Today,  should  provide  detail  about  today’s  status  and  indicate  why  the 
alert  was  the  color  it  was.  The  need  was  also  expressed  to  display  a  clear  indication  of  the  source  of  a 
link.  Representative  comments  were: 
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Summary  Page 

•  An  initial  problem  was:  “what  should  be  on  . . .  page?” 

•  It’s  important  to  communicate  and  have  a  common  understanding  of  what  should  be  on  the 
wall. 

•  For  ISR  [summary  page],  instead  of  today,  tomorrow,  and  long  range,  I  want  to  see  air,  land, 
surface,  subsurface  with  links  to  C2PC. 

•  [Discussing  CJTF  page]  We  need  to  include  things  like  the  commander’ s  intent.  Today  (what 
we  plan  to  do  for  next  10  days) . . .  what’s  going  on  today,  tomorrow,  etc. 

•  We  didn’t  ever  define  what  today,  tomorrow,  long  range  was  . . .  Sea  Control  had  a  good 
method:  Today  equals  what  I’m  going  to  do  today,  intentions,  etc. 

•  [Regarding  status  alerts]  We  need  to  understand  what  each  rule  set  is  and  how  it  is  being  used 
by  each  cell  and  document  it  for  the  CJTF  and  CINC.  The  rules  can  be  different  for  each  FA  as 
long  as  the  CINC  and  CJTF  know  the  rules. 

•  That’s  what  we  want  to  see:  what  they’re  planning  to  do  today,  what  are  their  intentions  for 
tomorrow.  .  .  . 


Links 

•  When  you  hit  a  button,  (today,  tomorrow,  long  range)  there  should  be  an  indication  of  what’s 
going  on. 

•  When  there’s  a  yellow  or  red  alert,  they  should  tell  you  what  they’re  doing  about  it. 

•  When  you  change  the  color  of  the  alert,  we  need  to  know  why  the  alert  changed. 

•  That’s  what  we  want  to  see:  what  they’re  planning  to  do  today,  what  are  their  intentions  for 
tomorrow. . . . 

•  [CJTF  page]  The  CJTF  should  provide  the  CINC  with  a  picture  of  what’s  going  on  so  he  can 
let  others  know. . . . 

•  [The  text  on  a  link  from  a  yellow  status  alert  included  a  description  of  why  the  text/alert  was 
yellow]  They  should  all  do  that! 

•  It’s  a  problem  that  pages/links  don ’t  indicate  where  they  came  from . . .  need  to  include 
addresses  or  headings  on  links. 

New  content  areas.  Although  the  summary  pages  from  the  13  functional  areas  provided  useful 
information  to  the  K-Wall  users,  there  were  requests  that  other  types  of  information  be  represented  in 
the  same  format  to  be  displayed  on  the  K-Wall.  During  the  course  of  the  game,  requests  to  provide 
such  information  led  to  the  development  and  display  of  summary  pages  with  the  following  content: 

•  CJTF — this  was  developed  and  updated  by  the  CJTF  staff  (BWCs,  with  guidance  from  the  sen¬ 
ior  staff)  and  was  originally  discussed  as  a  means  of  passing  CJTF  intent  to  junior  staff  but 
evolved  as  a  means  of  updating  more  senior  players  (CINC  and  NCA).  Once  implemented, 
much  discussion  focused  around  the  content  of  the  information  to  provide  on  the  suntunary 
page  and  link  to. 

•  Communications  (Blue  Comms) — this  included  information  related  to  local  (NWC) 
communication  status,  as  well  as  that  the  communication  capability  related  to  the  game. 

•  Other  tools  that  were  displayed  on  the  K-Wall — CAESAR,  TAPS,  JFLEX. 

The  development  of  such  pages  allowed  information  from  these  sources  to  be  integrated  with  the 
K-Wall  in  a  consistent  manner,  and  provided  users  with  the  same  features  that  the  functional  area 
summary  pages  did  (drill-down,  consistent  format  etc.).  Other  information  that  was  requested  and, 
when  available,  displayed  on  the  K-Wall  was  that  related  to: 
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•  METOC  (requested  and  displayed) 

•  Blue’s  Red  Cell  information  (requested) 

•  A  searchable  reference  library — maps,  ROEs,  capabilities,  etc.  (requested) 

3.2.2.  Summary  Page  and  Links  Format 

The  most  prevalent  suggestion  with  respect  to  the  sum¬ 
mary  page  format  was  to  dedicate  more  room  to  graphical 
information  and  less  room  to  text  information.  This  repre¬ 
sents  both  the  acknowledgement  of  the  usefulness  in  repre¬ 
senting  information  in  a  graphical  format  as  well  as  the 
difficulty  in  reading  the  text  on  the  summary  pages  (see 
Visibility  discussion  below,  section  3.3.2).  The  comments 
relating  to  the  format  of  the  links  from  the  summary  pages 
reflected  the  same  acknowledgement  and  stressed  the  need  to 
“keep  it  simple.”  K-Wall  users  explicitly  requested  that 
graphics  be  used  as  much  as  possible  and  when  not  possible,  simple,  large  black  and  white  text  be 
presented.  Comments  relating  to  the  format  of  summary  pages  and  links  include: 

Summary  Page 

•  Suggested  changing  the  summary  page  layout  so  that  the  entire  page  is  a  picture  with  small 
window  of  alerts/links  in  comer. 

•  There’s  too  little  room  on  the  summary  page. 

Links 

•  When  graphics  link  to  PowerPoint  slides,  use  simple  text,  no  logos,  no  pictures,  use  black  and 
white  text . . .  the  links  are  just  [too]  full  of  stuff. 

•  ...  too  much  data,  need  simple  PowerPoint  slide. 

•  We  need  standards  of  use  (e.g.,  colors  used,  processes  completed,  level  of  detail,  purposes, 
etc.). 

•  You  need  to  put  the  most  important  information  on  the  first  slide. 

•  If  they  put  a  lot  of  text  on  it,  I  just  skip  right  over  it  [when  monitoring  the  new  links^.  The 

information  needs  to  be  condensed. 

•  Keep  things  simple,  use  big  bullets. .  .  . 

•  A  business  rule  should  be:  don’t  use  too  many  acronyms  because  not  everyone  understands 
them. 

•  Italicized  text  on  the  link  is  hard  to  read. 

•  Yellow  text.  .  .  .  hard  to  read  on  white  background. 

•  If  a  whole  page  of  text  was  on  the  K-Wall,  we  didn’t  even  read  it. 

•  Too  much  detail ...  if  you  had  a  picture  showing  the  same  thing,  that  would  be  great  [list  in 

Word  doc,  small  font}. 

•  They  are  not  using  TacGraph  enough,  they  should,  because  there’s  not  a  lot  of  space  on  a  page 
...  a  graphic  would  have  told  ten  thousand  words. 
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3.2.3.  Summary  Page  Update 

Even  with  the  right  content  and  format,  the  summary 
pages  are  not  useful  unless  they  are  updated  at  an  adequate 
rate  to  reflect  events  and  changes  in  the  operational  situation. 

Early  in  the  game,  user  comments  suggested  that  the  K-Wall 
users  believed  that  the  summary  pages  were  not  being 
updated  at  a  sufficient  rate.  However,  this  perception 
appeared  to  wane  as  the  game  progressed  and  the  producers 
of  the  summary  pages  became  more  proficient  at  creating 
summary  pages  and  links.  Examples  of  these  comments  are: 

•  Some  of  the  dots  have  changed  color  but  they  still 
don’t  have  links  associated  with  them. 

•  Is  the  game  paused?  Why  is  nothing  updating? 

•  And  later  in  the  game  . .  .  The  game  is  progressing  smoothly  ...  the  K-Wall  is  getting  updated 
regularly. 

An  indication  of  whether  the  summary  pages  conveyed  the  “right”  information  to  users,  as  well  as 
how  well  the  summary  pages  were  being  updated,  comes  from  an  examination  of  whether  the  con¬ 
tents  of  the  K-Wall  reflected  the  important  events  of  the  game.  For  Global  2000,  information 
providers  (many  times  the  same  ones  that  updated  the  K-Wall  summary  pages),  game  personnel,  and 
others  provided  information  about  game  developments  by  posting  to  the  Sitsum  file  located  within 
WIGS.  This  document  was  essentially  a  list  of  text  describing  events,  the  time  of  each  event  and  the 
source  of  this  information  (i.e.,  who  posted  the  event  to  the  Sitsum).  It  was  ordered  according  to  the 
time  that  items  were  posted,  and  it  consisted  of  hundreds  of  postings  every  day. 

The  K-Wall  users  could  access  the  Sitsum  (via  WIGS  on  the  K-Wall),  to  become  notified  about  the 
most  recent  game  events — i.e.,  they  could  pull  information  from  WIGS.  This  often  involved  scrolling 
down  to  read  items  that  had  been  posted  more  than  a  few  minutes  before  the  current  time.  However, 
if  the  K-Wall  summary  pages  were  being  updated  at  a  sufficient  rate,  the  most  important  of  these 
events  should  also  be  reflected  in  the  K-Wall  content — ^i.e.,  information  providers  should  be  pushing 
information  to  the  K-Wall  via  the  summary  pages.  An  analysis  was  conducted  that  compared  the 
percentage  of  time  that  important  events  were  communicated  on  the  K-Wall  in  these  two  ways  based 
on  information  from  the  screen  captures  of  the  K-Wall  (which  were  recorded  every  5  minutes)  and 
the  Sitsums.  The  Sitsums  and  the  critical  event  logs  from  each  day  of  the  game  were  examined  by  a 
Subject  Matter  Expert  (SME),  who  identified  37  of  the  most  important  developments  during  the 
game.  Another  SME  assigned  a  “criticality”  rating  to  these  events,  (1  =  critical  event,  2  =  significant 
event,  3  =  important  event,  4  =  event  of  interest.)  The  screen  captures  recorded  before  and  after  these 
events  were  then  examined  to  determine  if  the  contents  of  the  K-Wall  gave  an  indication  of  them  (on 
the  summary  pages  or  links,  on  the  Sitsum  displayed  on  the  K-Wall,  or  via  some  other  means,  such 
as  e-mail).  Over  70%  of  the  important  events  were  reflected  in  the  screen  captures  of  the  K-Wall  in  at 
least  one  format  and  32%  of  them  were  indicated  on  both  the  Sitsum  and  on  a  summary  page/link.'* 
The  mean  percentage  of  time  that  these  events  were  reflected  on  the  K-Wall,  as  a  function  of 
criticality  and  format  is  shown  in  Figure  14  and  the  amount  of  time  it  took  for  the  events  to  be 
reflected  (in  minutes,  after  the  event  was  posted  to  the  Sitsum)  is  shown  in  Figure  15. 
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^  Because  the  screen  captures  were  recorded  every  5  minutes,  these  numbers  are  likely  underestimates  of  the  actual 
percent  of  time  that  the  important  events  were  reflected  on  the  K-WalL 
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Percentage  of  Critical  Events  Indicated  on  KW  as 
Function  of  Source  and  Criticality 
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Figure  14.  The  percentage  of  critical  events  displayed  on  the  K-Wall  as  a 
function  of  the  format  source  of  the  information  and  the  criticality  of  the  event 
(1  =  critical  event,  2  =  significant  event,  3  =  important  event,  4  =  event  of 
interest). 
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Time  to  Reflect  Critical  Events  on  KW  as  Function  of 
Source  and  Criticality 
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Figure  15.  The  average  amount  of  time  it  took  for  critical  events  to  be  displayed 
on  the  K-Wall  as  a  function  of  the  format  source  of  the  information  and  the 
criticality  of  the  event  (1  =  critical  event,  2  =  significant  event,  3  =  important  event, 

4  =  event  of  interest). 

As  expected,  more  important  events  were  more  likely  to  be  displayed  on  the  K-Wall  (100%  for 
critical  events,  73%  for  significant  events,  67  %  for  important  events,  and  33%  for  events  of  interest). 
However,  when  they  were  displayed,  the  time  it  took  to  be  displayed  in  one  format  on  the  K-Wall  did 
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not  differ  as  a  function  of  criticality  (8.5  minutes,  8  minutes,  10.7  minutes,  and  9.5  minutes  on  aver¬ 
age).  Important  events  were  displayed  on  the  Sitsums  59%  of  the  time  after  8.7  minutes  vs.  37  %  of 
the  time  after  15.8  minutes  on  a  summary  page  or  link.  This  result  suggests  that  pulling  information 
from  the  Sitsums  was  a  faster  and  more  reliable  way  of  becoming  informed  about  critical  events  than 
relying  on  the  K-Wall  summary  pages  to  provide  this  information. 

3.2.4.  Summary  Page  Tools 

Alerting  Change  to  Summary  Pages 

One  of  the  most  prevalent  needs  that  emerged  from  the  obser¬ 
vational  data  was  for  tools  that  alerted  the  user  to  changes  to  the 
summary  pages.  K-Wall  users  had  great  difficulty  detecting 
when  changes  occurred  to  information  on  the  summary  pages 
and,  when  they  did,  determining  specifically  what  information 
changed  and  how  important  the  change  was.  The  need  for  a 
change  alerting  system  was  underscored  by  the  many  “fixes” 
that  were  improvised  and  implemented  to  alleviate  this  problem. 

Because  the  summary  pages  were  not  originally  designed  to 
provide  a  mechanism  for  alerting  change  to  K-Wall  users,  the  wall  users  and  information  producers 
introduced  business  rules  and  took  advantage  of  the  coding  used  by  html  to  indicate  “visited”  links  to 
help  them  become  apprised  of  changed  information.  The  specific  fixes  observed  during  Global  2000 
were: 


•  The  KMs  introduced  a  business  rule  early  in  the  game,  requiring  that  information  providers 
include  an  asterisk  to  a  new  link  and  that  the  BWCs  inform  (via  e-mail)  these  information  pro¬ 
viders  when  the  link  was  read  by  the  K-Wall  users  (this  also  relates  to  the  feedback  issue 
discussed  below).  The  information  providers  were  also  required  to  put  an  exclamation  point 
next  to  hyperlinks  that  they  believed  needed  to  be  addressed  immediately.  These  business  rules 
were  effective  in  alerting  the  K-Wall  users  to  changed  information,  but  the  rules  were  only 
adopted  by  some  of  the  information  providers  and  appeared  to  fall  into  disuse  later  in  the 
game. 

•  The  operator  of  the  K-Wall  typically  looked  for  the  standard  html  color  change  (from  purple  to 
blue,  and  vice  versa)  in  text-based  hyperlinks  to  determine  whether  a  link  had  been  accessed. 

A  problem  with  this  method  was  that  many  times  information  providers  made  changes  to  the 
underlying  documents  that  were  being  linked  to  and  not  to  the  summary  pages  themselves. 
[This  type  of  change  does  not  cause  a  hyperlink  to  change  color;  hence  it  led  to  some  confu¬ 
sion.] 

•  Information  providers  sent  change  notifications  (i.e.,  when  a  change  was  made  to  the  summary 
pages)  to  the  BWC  and  other  K-Wall  users  via  e-mail,  chat  windows,  phone  calls,  or  by 
walking  up  to  the  ICC  and  informing  them. 

•  Information  providers  used  the  designated  graphic  area  on  the  summary  page  as  an  alert  win¬ 
dow  (e.g.,  using  the  text  “alert”). 

Comments  and  observations  relating  to  this  need  for  alerting  change  included: 

•  Change  the  stop  lights  so  they  flash  when  changed,  like  the  gauges  on  WIGS. 

•  There  are  problems  .  .  .  with  automatic  alerts  and  things  like  that. .  .  . 

•  When  they  update  their  page  they  need  to  update  their  links  [problem  with  new  links  not  being 
blue]. 
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•  ...  there  isn’t  anything  flashing  to  let  me  know  it’s  changed. 

•  On  the  K-Wall,  the  overview  stoplight  graph  does  not  automatically  reflect  the  status  of  the 
various  summary  slides  when  they  are  updated. 

•  I’m  looking  for  blue  text  as  an  indication  of  new  information. . .  . 

•  When  I’m  looking  for  pages,  I  don’t  use  the  overview  slide  ...  I  look  for  blue  links  and 
asterisks. 

Feedback  to  Information  Providers 

Also  emerging  as  a  needed  tool  for  K-Wall  users,  is  one  that  would  enable  them  to  inform  infor¬ 
mation  providers  that  a  particular  page  or  link  had  been  accessed,  and  convey  their  information 
needs.  This  tool  would  clearly  benefit  the  information  pushers  since  they  would  know  when  their 
information  had  been  viewed,  and  their  comments  reflected  this  need.  Providing  this  type  of  feed¬ 
back  might  also  improve  the  update  rate  of  the  K-Wall  since  the  information  providers  would 
become  more  aware  of  how  often  their  products  were  being  used.  The  CJTF  summary  page  that  was 
developed  during  Global  2000  was  one  means  to  convey  the  information  needs  of  the  K-Wall  users 
to  the  information  providers;  however,  there  was  no  tool  available  that  indicated  to  the  information 
providers  what  the  K-Wall  users  had  looked  at  or  drilled  down  to  using  the  K-Wall.  Comments 
reflecting  the  need  for  this  tool  were: 

•  We  need  a  way  for  K-Wall  users  to  convey  desires/disseminate  information  “down”  to  anchor 
desks  and  lower  echelons. 

•  [Information  provider  in  the  same  room  as  JCC]  I  have  an  advantage  over  people  downstairs 
because  I  can  walk  over  and  see  if  they’ ve  looked  at  my  links  or  not  [using  color  of  link  text] . 

•  Give  the  admirals  a  “laser”  and  check  boxes  that  they  can  check  once  they’ve  read  something. 

•  There  needs  to  be  automatic  feedback  that  the  page  has  been  read. 

•  There  needs  to  be  a  feedback  loop  on  K-Wall  from  BWC  to  AD/FA. 

•  Feedback  notification  is  required  when  something  urgent  has  been  received  by  the  CJTF. 

•  I  recommend  that  the  signal  be  cleared  once  the  commander  views  the  status. 

3.3.  K-WALL  USAGE  AND  USABILITY 

An  important  objective  of  the  Global  2000  K-Wall  data  collection  effort  was  to  examine  how  the 
K-Wall  was  used  during  a  realistic  operation  and  to  identify  any  usability  issues  related  to  its  use  and 
operation.  This  section  of  the  results  discusses  the  observed  usage  of  the  K-Wall,  including  usage 
context  (how  and  when  the  JCC  staff  used  the  K-Wall)  and  operation  (methods  employed  by  the 
K-Wall  operator)  and  identifies  several  problems  related  to  the  K-Wall  usability  and  visibility. 

3.3.1.  Usage  Context:  When  and  How 

The  K-Wall  was  used  in  a  variety  of  contexts  during  Global  2000. 

The  most  common  way  in  which  it  was  used  was  as  a  “situation 
assessment”  tool.  JCC  staff  (and  others)  continuously  viewed  the  K- 
Wall  throughout  the  game  to  maintain  SA  and  to  discuss  information 
with  each  other.  Many  user  comments  reflected  this  and  also 
highlighted  its  importance  in  eliminating  the  need  for  these  personnel 
to  be  briefed  daily  by  the  component  commanders.  The  K-Wall  was 
also  used  as  a  briefing  tool,  especially  during  the  transition  between  operational  phases  of  the  game. 
Observations  and  example  user  comments  included: 
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•  The  K-Wall  was  used  for  briefing,  discussion  of  information  without  having  to  stop  the  war. 

•  We  didn’t  do  a  briefing  at  8  o’clock,  1  o’clock  ...  we  didn’t  need  to  be  brought  up  to  date 
because  we  were  always  up  to  date  . .  .  we  didn’t  have  to  go  off  for  2  hours  and  prepare  a  brief. 

•  The  K-Wall  was  used  as  a  “situation  assessment  tool,”  continuously  viewed  by  JCC  staff. 

•  The  K-Wall  was  used  as  briefing  tool  by  component  commanders  to  individual  or  groups  of 
JCC  staff. 

•  ...  a  good  tool  to  transition  in  and  out  of  briefing  and  for  [assessment]. 

•  The  K-Wall  was  used  to  review  information  to  put  on  the  CJTF page. 


3.3.2.  Visibility  and  the  K-Wall 

Many  user  comments  concerned  the  diffi-  jL 

culty  that  K-Wall  users  had  in  seeing  informa-  —  -if  T  ^ 

tion  on  the  K-Wall.  These  comments  primarily 

reflected  problems  in  reading  the  text  and  small  ^  i/ 

objects  and  controls  displayed  on  the  smaller  ^  ^0^ 

peripheral  monitors.  This  difficulty  was  also 

reflected  by  the  fact  that  the  K-Wall  operator  “  "  ' 

typically  brought  information  from  the  periph¬ 
eral  monitors  into  focus  on  one  of  the  central  monitors  in  order  to  read  it.  However,  even  the  infor¬ 
mation  on  the  focus  monitors  was  sometimes  found  to  be  difficult  to  read  when  it  was  too  small  or 
when  other  K-Wall  users  were  obscuring  the  view  of  the  monitors.  Representative  comments  related 
to  the  K-Wall  visibility  included: 


•  It’s  hard  to  see  items  in  pull  down  menus  when  you’re  trying  to  bring  up  a  new  summary  page 
[in  monitor  11]. 

•  Here’s  another  part  of  the  K-Wall:  you’ve  got  to  have  good  eyes. 

•  The  K-Wall  operator  frequently  walked  up  to  monitor  9  and  10  to  read  them. 

•  K-Wall  users  occasionally  printed  pictures  to  look  at  them  (more  common  on  Coronado). 

•  He  [the  CJTF]  couldn’t  see  the  Strike  page  so  he  brought  it  up  on  his  laptop. 

•  Whenever  I  say,  “hey  COS  look  at  this  on  the  wall,”  he  has  to  get  out  his  glasses. 

•  Can  you  see  that  tab?  [on  bottom  of  Excel  page  on  M9,  other  BWC’s  head  blocking  his  view]. 

•  With  all  the  monitors,  needing  to  focus  eyes  is  a  problem.  If  people  have  a  headache  they  can’t 
make  decisions. 

•  Many  of  the  monitors  are  too  far  away  to  be  seen  easily  in  JCC. 

In  order  to  assess  the  difficulty  in  viewing  information  on  the  summary  pages  on  the  small  periph¬ 
eral  K-Wall  monitors,  a  “vision  test”  using  the  summary  page  format  was  conducted.  Three 
observers  with  normal  vision  (unaided  or  corrected  to  20/20)  were  presented  summary  pages  that  had 
the  same  layout  and  format  as  those  displayed  on  the  K-Wall  during  Global  2000,  but  that  had  been 
populated  with  random  text.  An  example  stimulus  is  shown  in  Figure  16.  They  were  asked  to  identify 
all  of  the  text  characters  displayed  on  the  page.  Accuracy  at  identifying  the  text  was  measured  when 
the  observers  were  seated  at  each  of  four  positions  on  the  JCC  command  table  (positions  A-D,  see 
Figure  16),  for  summary  pages  that  were  displayed  at  the  most  central  monitor  (Monitor  4)  and 
furthest  (with  respect  to  the  four  positions)  monitor  (Monitor  1). 
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Figure  16.  Left:  The  positions  (A,  B,  C,  D)  from  which  measurements  were  taken  and  the  monitors 
(1,4)  on  which  stimuli  were  presented  for  the  “vision  tests.”  Right:  Example  summary  page  stimulus 
used  in  vision  tests. 

Identification  accuracy,  averaged  across  all  observers,  as  a  function  of  distance  and  viewing  angles 
for  each  of  the  four  viewing  positions  is  given  in  Table  2.  As  is  shown,  perfect  identification  of  all 
the  characters  was  only  approached  when  a  summary  page  was  displayed  on  Monitor  4.  Viewers  had 
much  more  difficulty  reading  the  summary  page  text  on  Monitor  1 ,  especially  at  smaller  viewing 
angles. 


Table  2.  Mean  percent  correct  for  identification  of  characters  on  vision  test  stimuli 
when  seated  at  the  four  command  table  positions. 


Position 

Distance 

(inches) 

Angle 

(degrees) 

Percent  identified 

Monitor  1 

Monitor  4 

A 

147 

90 

92 

95 

B 

137 

70 

77 

98 

C 

98 

61 

85 

98 

D 

69 

52 

81 

100 

Table  3  gives  the  same  data,  broken  down  by  text  size  and  collapsed  across  viewing  positions. 
Identification  accuracy  for  the  time  and  date  and  the  bulleted  alerts,  impacts,  and  links  (level  5)  was 
very  low  when  the  page  was  presented  on  Monitor  1.  This  has  important  implications  because  this  is 
the  text  of  the  hyperlinks  to  the  alerts,  impacts,  and  links  pages,  which  were  updated  and  accessed 
very  frequently. 
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Table  3.  Mean  percent  correct  for  identification  of  characters  on  vision  test  stimuli 
for  the  different  summary  page  text  sizes. 


Text 

Font  Size 

Percent  identified 

Monitor  1 

Monitor  4 

1  (title) 

48  point 

97 

99 

2  (status  text) 

40  point 

97 

100 

3  (“alerts,”  “impacts,”  “links”) 

40  point 

100 

100 

4  (date) 

24  point 

70 

99 

5  (bullets) 

20  point 

61 

92 

The  above  data  corroborate  the  comments  and  behaviors  of  the  K-Wall  users  that  were  observed 
and  suggest  that  visibility  was  problem  with  the  K-Wall  during  Global  2000. 


3.3.3.  Use  of  K-Wali  Operations 

The  K-Wall  application  provided  several 
operations  that  allowed  users  to  configure  and 
reconfigure  the  K-Wall  content,  update 
information,  and  run  other  tools  in  conjunction 
with  the  K-Wall.  The  menu  bar  (see  Figure  17) 
on  each  window  included  controls  common  to 
Web  browsers  (Back,  Forward,  Refresh,  and 
Home  buttons,  an  address  bar,  a  menu  of  favorite 
Channels/Functional  Area  Summary  Pages),  and  buttons  that  allowed  peripheral  monitors  to  be 
brought  into  focus  in  one  of  the  two  central  displays  (Focus  Left  (9)  and  Focus  Right  (10)).  A  control 
menu  (see  Figure  1 8),  accessible  via  a  button  on  each  window,  allowed  users  to  hide  or  show  under¬ 
lying  applications  (e.g.,  C2PC,  Outlook,  Word,  etc.)  and  gave  users  access  to  four  preset  configura¬ 
tions  of  the  windows.  The  four  presets  were: 

1.  all  windows  shown  except  9  (left  focus  window)  and  10  (right  focus), 

2.  all  windows  shown  except  8  (bottom  left  windows)  and  9, 

3.  all  windows  shown  except  10  and  1 1  (bottom  right  windows), 

4.  all  windows  shown  except  8,  9,  10,  and  1 1  (all  lower  windows). 


The  purpose  of  these  presets  was  to  support  the  display  of  tactical  information  in  multiple  K-Wall 
displays,  at  least  one  of  which  was  a  focus  window. 


t 
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Figure  17.  The  K-Wall  menu  bar. 
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Figure  18.  The  K-Wall  control  menu. 

The  usage  of  a  subset  of  these  operations  (Home,  Refresh,  Focus  9,  Focus  10,  Hide,  Show,  Presets) 
was  included  as  information  in  the  data  logs.  Figure  19  shows  the  number  of  these  K-Wall  operations 
performed  per  hour  as  a  function  of  operational  phase. 


Figure  19.  The  K-Wall  operations  performed. 

As  can  be  seen,  the  most  commonly  performed  operations  were  Focus  9  and  Focus  10  and  Refresh. 
The  use  of  the  focus  operations  reflects  the  observed  behavior  of  bringing  information  from  the 
smaller  peripheral  monitors  into  focus  so  it  could  be  examined  in  more  detail.  This  behavior 
increased  over  time.  The  increase  in  the  number  of  refreshes  was  most  likely  related  to  the  fact  that. 
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late  in  the  game,  the  rate  that  the  K-Wall  was  automatically  refreshed  was  changed  from  once  every 
1  minute  to  once  every  5  minutes®.  The  other  K-Wall  operations  were  performed  less  often.  The  hide, 
show,  and  home  functions  were  rarely  used,  and  the  presets  were  almost  never  taken  advantage  of. 
This  latter  finding  is  probably  due  to  the  fact  that  preset  1  (all  windows  shown  except  9  and  10)  was 
the  default  configuration  that  was  selected  when  the  K-Wall  software  was  started  each  morning  of 
the  game.  This  setting  allowed  for  C2PC  to  be  displayed  in  the  two  focus  monitors,  which  more  than 
adequately  met  the  requirement  for  the  K-Wall  to  provide  a  tactical  focus  (see  the  discussion  of  user 
requirements,  above). 

3.3.4.  Evolution  of  K-Wall  Operation/Control 

The  operation  and  control  of  the  operation  of  the  K-Wall 
changed  over  the  course  of  the  game  as  K-Wall  users  and 
potential  operators  became  more  familiar  with  it.  Prior  to 
game  start  and  for  the  first  several  hours  of  game  play,  the  K- 
Wall  operation  was  primarily  conducted  by  one  of  the  K-Wall 
designers/technology  support  staff.  At  this  stage,  wall 
operation  was  largely  directed  by  the  BWC,  with  some 
direction  coming  fi'om  the  rest  of  the  CJTF  staff.  Early  into 
the  game,  however,  the  BWC  increasingly  took  direct  control 
of  the  wall.  As  part  of  his  role  as  BWC,  he  assumed  responsibility  for  looking  for  changes  in  the 
peripheral  monitors,  managing  the  information  displayed  on  the  wall,  moving  the  information  from 
the  small  monitors  into  focus  on  one  of  the  large  displays,  and  drawing  the  attention  of  the  CJTF  staff 
to  important  changes  and  information.  Mid-game  (several  days  into  game  play),  other  CJTF  support 
personnel  began  to  operate  the  wall  and  became  more  proactive  with  regard  to  controlling  the 
contents  of  the  K-Wall.  Comments  made  during  and  following  the  game  reflected  acknowledgement 
by  wall  users  of  the  importance  of  a  K-Wall  operator  who  was  both  skilled  at  using  the  technology 
and  who  had  enough  operational  knowledge  to  manage  wall  content  effectively: 

•  [CJTF  to  K-Wall  operator,  BWC]  You  should  notice  when  changes  occur  and  bring  it  up  on 
the  wall. 

•  Sometimes  they  [CJTF  et  ah]  ask  me  to  bring  stuff  up  on  the  wall. 

•  There  was  an  evolution  from  them  looking  for  information  to  bring  up  on  the  wall  to  seniors 
requesting  information  that  they  knew  was  there. 

•  Knowing  what  and  where  information  is  located  (and  that  it  exists)  is  vital  to  making  the  wall 
useful. 

•  The  K-Wall  operator  evolved  into  a  Knowledge  Warrior.  .  .  . 

•  In  order  for  the  K-Wall  to  be  successful,  there  must  be  a  person  dedicated  to  operating  it. 


®  The  change  in  the  refresh  rate  was  due  to  a  business  rule  imposed  during  the  game  in  an  attempt  to  limit  demands 
on  available  bandwidth. 
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3.3.5.  Usability  Probiems,  Ease  of  Use 

Although,  in  general,  the  K-Wall  operators  and  viewers  found  the  K-Wall  easy  to  use,  several 
usage  problems  were  identified  based  on  observed  behaviors  and  comments.  K-Wall  operators  com¬ 
plained  that  accessing  information  that  was  found  in  locations  other  than  the  displayed  summary 
pages  or  their  links  was  difficult  because  it  involved  many  steps.  This  usually  required  navigating  to 
that  information  via  the  WIGS  system  or  via  the  K-Wall  channels  menu  (see  Use  of  K-Wall  Opera¬ 
tion  above,  section  3.3.3).  The  fact  that  input  to  the  K-Wall  came  from  two  sets  of  mouse  and  key¬ 
board  controls  plus  the  touch  screens  caused  some  confusion  when  more  than  one  person  attempted 
to  operate  the  K-Wall  simultaneously.  Confusion  also  arose  because  only  the  focus  monitors  had 
touch  screens.  Users  were  observed  attempting  to  use  the  screens  of  the  peripheral  monitors  as  if  they 
were  touch  screen-capable.  Another  commonly  observed  problem  was  the  tendency  for  K-Wall  users 
to  lose  the  mouse  cursor  in  so  much  screen  real  estate — especially  after  looking  away  from,  and  then 
returning  their  view  to  the  display.  Certain  design  features  of  the  K-Wall  were  intended  to  prevent 
this  problem;  users  were  provided  with  both  a  specially-enlarged  cursor  and  a  hot-key  combination 
that  highlighted  the  cursor  position  on  the  display.  K-Wall  users  however,  still  experienced  some 
problems  finding  the  mouse  cursor.  Comments  relating  to  K-Wall  usability  issues  included: 

•  With  less  than  2  weeks  of  use,  the  team  was  able  to  efficiently  use  the  K-Wall. 

•  Where’s  my  mouse?  [wiggling  the  mouse  to  find  the  cursor] 

•  It  was  found  to  be  useful  by  most  users.  .  .  . 

•  Navigating  using  the  channel  menu  is  inconvenient.  For  example,  getting  to  WIGS  involves 
many  steps. 

3.3.6.  Preference  and  Comparison  to  Other  Tools 

In  general,  K-Wall  user  comments  suggest  that  they  liked  the  K-Wall, 
though  users  believed  that  there  were  problems  that  needed  to  be  addressed 
and  various  improvements  that  could  be  made  to  its  design.  When  comparing 
the  K-Wall  to  the  tools  they  currently  had  available  to  them,  they  expressed  a 
preference  for  the  K-Wall.  Representative  comments  were: 

•  This  is  much  better  than  what  we  had  before,  better  than  a  stickpin  in  a 

•  The  K-Wall  was  a  hit. .  .  . 

•  It’s  a  great  information  filter,  but  there  are  problems.  .  .  . 

•  [Issue  of  defining  what  alerts  mean]  wouldn’t  be  important  if  they  didn’t  like  the  K-Wall,  but 
that’s  what  they’re  using. 

•  [Discussing  Global  2000  in  general]  We’re  just  light  years  ahead  of  last  year  ...  but  each  year 
should  get  better. 

•  [The  K-Wall]  is  a  good  idea  but  it  needs  some  tweaking. . . . 

•  The  CINC  had  more  negative  comments  . . .  [CINC  was  located  aboard  Coronado.] 

•  [From  AD  perspective]  ...  as  people  got  used  to  the  tools,  then  there  was  more  use,  more 
“let’s  put  this  on  the  wall”. . .  . 

•  The  K-Wall  was  a  good  first  step.  . . . 

•  Life  would  have  been  miserable  if  I  didn’t  have  this. .  .  . 

•  It’s  a  great  tool  you’ve  got  here,  don’t  get  me  wrong.  .  .  . 
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3.4.  OTHER  EMERGING  THEMES 

Although  the  current  data  collection  effort  was  narrowly  focused  in  scope,  intending  to  answer 
questions  related  to  K-Wall  information  requirements  and  usability  issues,  the  observational  data  also 
provided  important  insights  to  several  Knowledge  Wall-  and  Knowledge  Web-related  issues.  These 
other  categories  of  data  are  discussed  briefly  below.  In  particular  they  concern  questions  regarding: 

•  How  flexible/tailorable  should  the  Knowledge  Wall  and  Summary  Page  content  be  (with 
respect  to  different  users)  and  who  should  have  access  to  them? 

•  How  flexible/tailorable  should  the  Knowledge  Web  format  and  content  be  (with  respect  to 
different  users)? 

•  What  issues  are  related  to  the  production  and  update  of  summary  pages  (from  the  perspective 
of  information  providers)? 

•  How  was  the  K-Wall  on  Coronado  used  and  what  usability  or  other  issues  were  related  to  it? 

3.4.1.  K-Wall  and  K-Wall  Summary  Page  Access  and  Tailorable  K-Walls 

During  Global  2000,  the  information  providers  in  the  13  functional  areas  created  summary  pages 
with  the  intent  that  they  would  be  displayed  on  the  CJTF  K-Wall  in  the  JCC.  Therefore,  these 
summary  pages  contained  content  and  links  to  meet  the  specific  needs  of  the  CJTF.  However,  an 
additional  K-Wall  existed  on  Coronado.  Although,  the  users  of  this  K-Wall  were  the  CINC  staff,  the 
same  summary  pages  were  available  and  frequently  displayed  here.  No  summary  pages  were  created 
to  specifically  meet  the  information  requirements  of  the  CINC  (one  exception  to  this  was  the  CJTF 
page,  which  was  intended  to  provide  a  means  of  pushing  information  from  the  CJTF  up  to  the  CINC 
and  NCA).  In  addition,  the  summary  pages  were  available  to  all  Global  2000  players — ^they  could  be 
accessed,  via  the  WIGS  system,  on  any  player’s  console.  This  allowed  information  to  be  shared 
easily  across  the  different  functional  areas. 

Comments  made  by  the  K-Wall  users,  information  providers  and  others,  such  as  the  CINC  and 
NCA  staffs,  suggested  that  there  were  both  benefits  and  costs  associated  with  this  accessibility  of 
information.  On  the  one  hand,  the  information  providers,  especially,  found  it  useful  to  be  able  to 
view  the  information  being  posted  by  other  functional  areas  to  become  apprised  of  the  operational 
situation.  They  frequently  provided  links  from  their  own  summary  pages  to  information  produced  by 
others.  Many  comments  also  reflected  the  desire  of  the  information  providers  to  have  access  to  the 
CJTF  K-Wall  itself  (or  a  window  into  it),  in  order  both  to  access  the  information  from  other 
functional  areas  and  to  see  what  the  CJTF  was  looking  at. 

On  the  other  hand,  although  there  was  agreement  that  others,  such  as  the  CINC,  should  have  their 
own  K-Wall,  the  way  in  which  this  was  implemented  during  Global  2000  fell  short  of  players’ 
expectations.  This  perception  stemmed  from  the  fact  that  the  summary  page  information  available  to 
be  displayed  on  the  CINC  K-Wall  had  been  created  for  the  CJTF  and  was  insufficient  (and  inappro¬ 
priate)  to  meet  the  requirements  of  the  CINC.  As  one  user  stated:  “One  K-Wall  glove  does  not  fit 
all.”  A  great  deal  of  discussion  centered  around  the  idea  of  making  K-Wall  and  its  content  tailorable, 
to  fit  the  needs  of  its  specific  users.  Representative  comments  included: 

Access  to  Summary  Pages 

•  Another  important  feature  is  that  you  can  call  pages  up  on  other  machines. 
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•  We  [MARFOR]  checked  the  other  pages  periodically. 

Access  to  a  K-Wall 

1 .  Functional  area  personnel 

•  Several  cell  personnel  (in  the  same  room  as  JCC)  have  come  up  to  ask  where  the  ISR 
picture  (that  they  can  see  on  the  K-Wall)  is  located. 

•  Lower  echelons  need  some  way  to  tell  what  is  currently  being  displayed  on  the 
K-Wall. 

•  There  are  questions  on  whether  the  large  screen  displays  on  the  sides  of  JOC  can 
be  slaved  to  show  what  the  head  table  is  viewing  up  elose  on  the  K-Wall?  Since  the 
K-Wall  was  only  viewable  from  JCC,  I  don’t  believe  the  K-Wall  was  of  much  value 
[to  the  cells]. 

2.  CINC/NCA 

•  There  was  concern  expressed  by  the  JCC  staff  that  the  CINC  isn’t  seeing  everything 
they  can.  . . .  the  same  concern  was  expressed  by  the  CINC. 

3.  Tailorability  of  the  K-Wall 

•  Whose  tool  is  it?  If  three  groups,  CJTF,  CINC,  NCA,  can  see  the  K-Wall  they  need  to 
see  different  information  . . .  One  K-Wall  glove  does  not  fit  all — there  needs  to  be 
tailorable  information.  . .  . 

•  Do  we  give  everyone  access  to  all  the  K-Wall  and  hope  they  access  only  the 
information  they  should  or  restrict  access? 

•  The  CINC/NCA  should  have  customizable  version  of  the  K-Wall. 

3.4.2.  Issues  Related  to  Generating  and  Updating  the  Summary  pages 

The  requirement  to  create  and  maintain  summary  pages  was  perceived  initially  as  a  burden  by 
some  information  providers.  The  task  of  continuously  providing  updates  to  senior  staff  via  the  K- 
Wall  was  viewed  as  manpower  intensive  and  time  consuming,  requiring  at  least  a  single  individual  to 
be  dedicated  to  performing  it  for  a  significant  proportion  of  the  time. 

However,  this  was  offset  by  the  fact  that  many  information  providers  also  felt  that  updating  the 
summary  pages  was  less  demanding  than  the  tasks  that  they  would  typically  have  had  to  do  to  pre¬ 
pare  for  the  daily  briefings  to  seniors.  Information  providers,  by  and  large,  provided  links  from  the 
summary  pages  to  information  products  that  they  were  already  producing,  allowing  them  to  perform 
this  task  concurrently  with  others  and  eliminating  the  need  to  dedicate  hours  of  time  to  preparing  for 
the  briefing.  Comments  related  to  this,  which  were  made  by  information  providers  and  others, 
occurred  later  in  the  game,  after  they  had  become  more  familiar  with  the  SumMaker  tool  provided  to 
create  and  update  the  summary  pages.  This  suggests  that  part  of  the  burden  initially  perceived  by 
information  providers  in  updating  seniors  was  related  to  unfamiliarity  with  the  change  in  operations 
(i.e.,  the  cancellation  of  traditional  daily  briefs),  and  the  requirement  to  learn  new  software  (i.e.,  the 
SumMaker  software).  Representative  comments  included: 

•  If  you  really  wanted  to  do  this  right,  you’d  have  to  have  someone  updating  the  pages  full 

time. 


•  It  was  a  burden  for  echelons — but  more  so  than  preparing  the  daily  brief?  Some  thought  it 
was,  others  did  not!  More  so  in  first  week,  as  people  got  used  to  the  tools,  then  there  was 
more  use,  more  “let’s  put  this  on  the  wall.” 

•  The  use  of  the  K-Wall  is  manpower-intensive  in  that  many  people  were  involved  in  its  use — 
and  the  preparation  of  the  content. 

•  Updating  the  K-Wall  was  a  burden  in  the  beginning.  Bottom  line:  for  a  nearly  real  time  K- 
Wall,  a  dedicated  person  must  be  there.  If  not,  the  K-Wall  will  be  populated  when  the  AD 
can  get  to  it.  It  will  be  populated  with  information  that  is  not  timely,  but  a  placeholder  that 
appeases  higher  ups  and  looks  good. 

•  Well-manned  missions  areas  are  being  updated  faster  than  poorly  manned  ones. 

3.4.3.  K-Wall  Usage  on  Coronado 

Although  the  current  data  collection  effort  focused  primarily  on  the  K-Wall  at  the  NWC,  there 
was  also  opportunity  to  collect  data  on  the  usage  of  the  K-Wall  in  the  Crisis  Action  Center  (CAC) 
onboard  Coronado.  Data  were  collected  over  a  number  of  days  during  the  Global  2000  game.  Data 
collected,  and  discussed  below,  included  observations  that  related  to  K-Wall  configuration  and  usage 
and  comments  related  to  problems  encountered  by  K-Wall  users. 

Operators  of  the  K-Wall  on  Coronado  had  access  to  the  same  files  and  most  of  the  same  tools  that 
were  available  at  the  NWC.  Just  as  the  K-Wall  in  the  NWC  JCC  could  be  configured  for  usage  by  the 
CJTF,  the  K-Wall  aboard  Coronado  could  be  flexibly  configured  to  show  information  required  by 
the  CINC.  Nonetheless,  the  two  K-Walls  were  configured  in  very  similar  ways  (i.e.,  with  tactical 
information  in  the  focus  monitors  and  summary  pages  in  the  supplemental  monitors).  An  important 
difference  found  between  the  CINC’s  K-Wall  and  the  CJTF’s  K-Wall  was  that  the  CJTF  summary 
page  was  displayed  in  the  lower  left  peripheral  monitor  aboard  Coronado.  This  is  a  significant 
observation  because  it  demonstrates  that  the  CJTF  page  was  being  used  as  it  was  intended,  i.e.,  to 
push  information  to  the  CINC. 

There  were  also  many  similarities  in  the  usage  and  operation  of  the  two  K-Walls.  In  each  case, 
the  K-Wall  was  used  for  situation  assessment  and  briefing  and  it  was  generally  controlled  by  a  single, 
dedicated  operator.  Problems  identified  in  user  comments  associated  with  the  CINC  K-Wall  were 
those  related  to  the  alerting  of  changed  information  on  the  summary  pages,  the  visibility  of  the  infor¬ 
mation  on  the  K-Wall  and  in  the  time  lag  associated  with  drill  down  (due  to  the  fact  that  information 
had  to  be  pulled  from  the  NWC  across  a  network).  As  discussed  above  (see  Tailorable  K-Walls),  the 
CINC  personnel  did  not  believe  that  contents  of  the  K-Wall,  particularly  the  implementation  of  the 
color  coding  of  the  status  alerts,  was  appropriate  or  correct  for  the  CINC.  This  was  to  be  somewhat 
expected  since  the  summary  pages  had  been  created  to  meet  the  needs  of  the  CJTF. 
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4.  SUMMARY  AND  DISCUSSION 


The  current  data  collection  effort  had  several  objectives: 

•  assess  whether  the  K-Wall  sufficiently  met  the  user  requirements  identified  previously, 

•  determine  whether  there  were  new  requirements, 

•  establish  usage  patterns, 

•  determine  whether  there  were  usability  and  usage  issues  that  need  to  be  addressed. 

The  following  discussion  summarizes  the  results  and  suggests  future  directions  for  K-Wall 
research  and  development. 

4.1.  HOW  WELL  WERE  THE  14  INITIAL  REQUIREMENTS  MET? 

The  initial  implementation  of  the  K-Wall  was  intended  to  address  the  14  requirements  identified 
by  previous  interviews  with  potential  K-Wall  users  (Smallman  et  al.,  2001).  The  K-Wall  was 
generally  considered  to  be  a  success  in  terms  of  meeting — at  least  to  some  reasonable  extent — the  14 
identified  requirements.  This  assessment  is  borne  out  in  both  user  comments  and  in  the  results  of  the 
data  analysis.  Of  course,  certain  characteristics  of  the  wall  were  judged  more  successful  than  others. 
In  general,  both  the  automated  and  observational  data  suggest  that  the  K-Wall  was  “a  good  first 
step.”  A  summary  of  the  14  user  requirements,  the  K-Wall  design  capability  intended  to  meet  them 
and  an  evaluation  of  whether  or  not  the  K-Wall  met  each  design  requirement  is  given  in  Table  4. 
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Table  4.  How  the  results  bear  on  the  success  (or  failure)  of  meeting  the  14  user  requirements 
with  the  prototype  K-Wall  design. 


User  Requirement 

K-Wall  prototype  design 
capability 

Requirement 

met? 

Shared  SA 

Shared  display 

V 

Integrated  Information 

Co-located  summary  pages 

~ 

Intuitive  Graphical  Interface 

Format 

Graphical  presentation  when 
possible 

V 

Consistency 

Consistent  format  summary  pages 

V 

Tactical  Focus 

Ability  to  view  multiple  tactical 
displays 

V 

Supplemental  Information 

Summary  pages  on  peripheral 
displays 

V 

Mission  goals  and  objectives 

Text  document 

Anchor  Desk  Output 

Summary  pages  with  links  to  more 
info 

V 

Connectivity/Collaboration 

Collaboration  tools  (IWS) 

- 

Cognitive  Support 

Limited  output  from  nonintegrated 
cognitive  support  tools 

Flexible  Configuration 

Any  pages  viewable  in  any  display 

V/- 

_  .  Drill-Down 

Feature 

Multiple  scalable  views,  links  to 
more  info 

V 

Information  Age  and  Reliability 

- 

Tactical  Overlays 

Various  software  for  tactical 
graphic  presentation 

V 

The  co-location  on  a  common  display  of  a  set  of  consistently  formatted  summary  pages  that  were 
continuously  updated  led  to  a  feeling  among  senior  staff  of  improved  SA  and  a  reduced  need  for 
them  to  continuously  interrogate  junior  staff  for  information.  The  K-Wall  provided  a  tactical  focus, 
when  necessary,  as  well  as  providing  users  with  supplemental  information  from  the  functional  area 
anchor  desks.  The  K-Wall  could  be  flexibly  configured  to  meet  the  changing  operational  situation, 
and  it  allowed  for  drill-down  to  detailed  information.  The  TacGraph  support  application  that  was 
used  for  the  generation  of  integrated  readily  digestible  representations  of  tactical  and  other  data  was 
an  unqualified  success.  Users  appreciated  the  TacGraph-generated  intuitive  integrated  graphical  sum¬ 
maries  available  on  the  summary  pages.  They  also  liked  the  ability  to  interrogate  these  graphical 
summaries  via  embedded  hyperlinks  to  supplemental  information. 
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Future  K-Wall  implementations,  however,  will  need  to  be  designed  to  meet  the  user  requirements 
that  were  not  sufficiently  supported  by  the  initial  K-Wall  prototype.  For  example,  K-Wall  user 
responses  suggested  they  had  difficulty  integrating  information  on  the  K-Wall,  an  important  issue  in 
future  K-Wall  displays.  There  is  more  to  information  integration  than  mere  co-location,  and  there 
were  calls  for  further  data  fusion  and  representation  of  inter-relationships  across  functional  areas. 
Users  also  continued  to  express  a  desire  for  flexibility  in  configuring  the  display  to  meet  the  needs  of 
different  K-Wall  users.  The  requirement  for  tools  that  support  cognitively  challenging  tasks,  in  par¬ 
ticular  those  that  support  the  attention  management,  alerting,  and  collaboration  were  not  perceived  to 
be  adequately  met.  The  notification  that  new  information  had  been  posted  by  providers  for  viewing 
by  senior  staff  is  part  of  a  more  general  process  of  communication  and  multi-tier  collaboration  that 
was  highlighted  during  the  game.  Users  expressed  a  need  for  the  display  of  information  about  goals 
and  commanders  intent  and  about  information  age.  Information  providers  need  more  guidance  in  the 
production  of  summary  pages  and  associated  content  to  produce  relevant  information  that  really  adds 
value.  Improving  this  collaboration — either  through  modified  business  rules  and/or  through  the  dis¬ 
play  of  explicit  context-sensitive  CCIRs — will  ensure  that  future  information  providers  more  clearly 
understand  what  they  must  push  to  the  K-Wall  so  that  senior  staff  get  the  information  they  need  when 
they  need  it. 

4.2.  HOW  WAS  THE  K-WALL  USED? 

The  K-Wall  was  used  continuously  throughout  the  Global  2000  game.  The  fact  that  it  became  a 
focus  during  the  game — it  was  commonly  perceived  as  the  place  where  the  “action”  was — is  testa¬ 
ment  to  the  fact  that  it  became  the  primary  situation  assessment  tool  for  the  CJTF  staff  The  K-Wall 
provided  a  means  of  visualizing  mission-status  in  a  continuous  fashion,  eliminating  the  need  for  a 
traditional  8-hour  briefing  cycle.  Updating  summary  pages  that  were  continuously  available  to  the  K- 
Wall  users  (and  others)  replaced  the  task  of  preparing  briefings  by  the  information  providers  and 
allowed  them  to  perform  it  in  parallel  with  other  tasks.  The  K-Wall  was  occasionally  used  to  brief  K- 
Wall  users.  However,  this  use  of  the  K-Wall  was  done  on  a  more  impromptu  basis,  as  the  operational 
situation  demanded,  rather  than  according  to  a  set  schedule.  Usage  of  the  CINC  K-Wall  on  Coronado 
followed  a  similar  pattern. 

The  operation  and  control  of  the  K-Wall  was  typically  performed  by  a  dedicated,  operationally 
savvy  individual,  under  the  direction  of  the  other  users  of  the  CJTF  staff,  or  in  collaboration  with 
them.  The  operator  monitored  its  content  for  new  information  on  a  continuous  basis,  bringing  rele¬ 
vant  information  into  focus  on  the  central  monitors  when  necessary,  in  order  that  others  could  view 
and  discuss  it.  The  K-Wall  was  flexibly  configured  according  to  the  operational  situation,  but  almost 
always  displayed  some  number  of  summary  pages  from  the  13  functional  areas  or  links  from  these 
pages.  Tactical  information  and  other  tools  were  also  commonly  presented  in  the  focus  windows.  The 
Web-based  knowledge  management  approach  that  serves  as  a  cornerstone  if  the  walls  design,  was 
successful  in  supporting  the  needs  of  both  the  K-Wall  users  and  the  information  providers — ^the  most 
commonly  accessed  information  via  the  K-Wall  was  in  html  format. 

Several  usability  problems  associated  with  the  operation  of  the  K-Wall  were  identified.  These 
included  confusion  caused  by  the  existence  of  multiple  concurrent  inputs  to  the  K-Wall  (multiple 
mice  and  keyboards  and  touch  screen  input),  difficulties  in  navigating  to  information  via  the  channel 
menu,  and  problems  related  to  the  poor  visibility  of  the  information  and  controls  on  the  K-Wall. 
Despite  these  problems,  however,  operators  and  controllers  of  the  K-Wall  generally  found  it  easy  to 
use. 


45 


Summary  pages  from  the  1 3  functional  areas  (and  links  from  them  to  other  information  products) 
were  the  most  commonly  displayed  information  sources  on  the  K-Wall.  Tactical  information  and  other 
software  were  also  displayed  as  needed,  especially  on  the  large  central  focus  monitors.  Special-purpose 
summary  pages  presenting  other  information  were  also  created  during  the  game  in  response  to  K- 
Wall  user  requests. 

4.3.  WHAT  NEW  DESIGN  REQUIREMENTS  OR  CHANGES  ARE  NEEDED? 

The  results  discussed  above  identified  many  issues  that  suggest  changes  and  new  requirements  for 
the  K-Wall.  These  new  requirements  should  be  implemented  in  future  K-Wall  designs,  in  the  design 
of  the  summary  page,  or  in  the  form  of  summary  page  tools. 

4.3.1.  K-Wall  Design 

Subsequent  K-Wall  designs  must  address  the  issues  discussed  above  that  relate  to: 

•  Alerting  users  to  change  in  “hidden”  information  and  directing  users  attention  to  important 
information  currently  displayed  on  the  K-Wall  (Overview  page  design,  pointers,  etc.) 

•  Integration  of  the  K-Wall  with  other  tools 

•  The  visibility  of  information  on  the  K-Wall 

•  The  usability  of  the  K-Wall 

One  of  the  ways  that  the  current  K-Wall  attempted  to  alert  users  to  changes  in  “hidden”  displays 
was  through  the  use  of  the  Overview  page.  This  page,  which  was  displayed  in  the  top-left  monitor 
during  a  portion  of  the  game,  indicated  the  “worst  case”  value  of  the  of  the  status  alerts  (Today, 
Tomorrow,  or  Long  Range)  of  all  of  the  13  functional  area  summary  pages,  including  those  that  were 
not  currently  being  displayed  on  the  K-Wall.  It  provided  links  to  each  of  these  sununary  pages.  This 
allowed  K-Wall  users  to  see  whether  changes  in  status  had  occurred  in  any  of  the  functional  areas 
and  to  compare  the  status  of  all  the  functional  areas  “at  a  glance.”  In  practice,  however,  the  Overview 
page  was  rarely  used  by  K-Wall  users  in  this  way.  Further,  it  was  not  even  displayed  for  almost  an 
entire  phase  of  the  game.  One  highly  likely  reason  for  this  was  that  changes  in  the  overview  were  not 
automatically  linked  to  changes  that  occurred  in  the  summary  pages  it  represented.  Instead,  the 
Overview  pages  had  to  be  updated  manually,  and  there  was  usually  a  time  lag  between  when  changes 
were  made  in  the  summary  pages  and  when  they  were  reflected  on  the  Overview  page.  Further,  the 
Overview  page  indicated  a  single  value  to  represent  all  three  of  the  summary  page  alerts,  so  users  did 
not  know  when  that  status  value  had  meaning  (today  vs.  tomorrow  vs.  long  range  vs.  all  of  these)  and 
they  could  not  answer  questions  such  as  “what  is  the  predicted  long  term  status  of  all  the  functional 
areas.”  A  redesigned  Overview  page  should  be  incorporated  into  future  K-Wall  designs,  which 
should  give  users  a  clear  indication  of  changes  over  time  and  provide  integrated  information  to  allow 
them  to  answer  questions,  such  as: 

•  What  is  the  current  status  of  the  different  functional  areas  and  how  and  when  might  this 
status  change?  (i.e.,  represent  status  over  time). 

•  Are  there  any  important  events  or  problems  that  need  to  be  addressed?  (i.e.,  alert  users  to 
problems  or  events  that  might  effect  status)  When  will  these  events  or  problems  have  an 
effect? 

•  Have  any  important  changes  occurred  in  the  content  of  the  summary  pages? 
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•  How  do  any  changes,  events,  or  problems  in  one  functional  area  have  an  effect  on  the  status 
of  other  functional  area(s)? 

Future  K-Wall  designs  should  also  take  into  account  the  need  for  tools  that  support  the  direction  of 
users’  attention  to  important  information  that  is  currently  displayed  on  the  K-Wall.  When  K-Wall 
users  wanted  to  discuss  or  brief  information  on  the  K-Wall  they  frequently  used  a  laser  pointer  or  the 
K-Wall  mouse  to  direct  other’s  attention  to  that  information.  This  behavior  suggests  the  need  for  a 
similar  pointing  device  to  be  incorporated  into  the  K-Wall  software.  The  fact  that  the  K-Wall  is 
intended  for  use  by  multiple  viewers  suggests  that  some  multi-user  pointer  might  be  valuable  for 
briefing  or  discussing  information  on  the  K-Wall.  However,  a  more  difficult  issue  that  future  designs 
need  to  take  into  account  is  how  to  direct  user’s  attention  to  information  on  the  K-Wall  that  they  are 
not  aware  needs  to  be  attended.  This  issue  is  best  addressed  in  the  design  of  the  summary  pages, 
which  are  discussed  below. 

An  obvious  solution  to  the  visibility  problems  of  the  K-Wall  would  be  to  alter  its  design  so  that  it 
comprises  fewer,  larger  monitors.  Increasing  the  size  of  the  displays  would  facilitate  easier,  more 
accurate  viewing  of  text,  graphics,  and  user  controls.  However,  there  is  a  trade-off  with  this  type  of 
design  solution;  it  may  introduce  problems  associated  with  reducing  the  number  of  types  of  informa¬ 
tion  displayed  at  one  time  on  the  K-Wall.  These  include  the  attention  issues  already  associated  with 
the  current  K-Wall  design,  such  as  how  to  alert  users  of  changes  in  the  information  that  is  not  cur¬ 
rently  displayed  on  the  K-Wall.  It  also  creates  problems  associated  with  deciding  what  information 
should  be  displayed  on  the  K-Wall  monitors  and  how  best  to  configure  that  information.  Further, 
navigating  to  information  not  currently  displayed  on  the  K-Wall  will  become  more  difficult  as  the 
ratio  of  information  windows  to  physical  monitors  increases  and  some  means  of  supporting  ease  of 
navigation  should  be  implemented.  For  example,  the  workspace  control  keys  evaluated  by  St.  John  et 
al.  (1999)  could  be  considered. 

Usability  issues  related  to  the  operation  of  the  K-Wall  included  those  that  were  observed  when 
multiple  users  attempted  to  control  the  K-Wall  simultaneously.  The  K-Wall  prototype  used  at  the 
Global  2000  War  Game  could  be  controlled  via  two  sets  of  mouse  and  keyboard  inputs,  as  well  as 
through  the  touch  screen  capabilities  of  the  focus  monitors.  This  occasionally  led  to  confusion,  such 
as  when  one  user  attempted  to  control  the  K-Wall  cursor  with  the  mouse  and  another  user  interfered 
by  touching  one  of  the  focus  monitors  to  access  a  link.  Future  K-Wall  designs  should  consider  how 
to  allow  multiple  users  to  effectively  control  the  K-Wall  and  how  control  can  be  shared  and  switched 
among  users  without  confusion.  Solutions  to  this  problem  might  be  incorporated  into  the  K-Wall 
software  but  might  also  involve  defining  business  mles  related  to  K-Wall  operation. 

4.3.2.  Summary  Page  Design  and  Links 

The  summary  pages  were  designed  to  provide  users  a  means  to  rapidly  acquire  and  integrate 
information  by  providing  them  with  a  high-level,  inmitively  presented  summary  from  the  various 
functional  areas.  They  provided  a  window  into  the  “Knowledge  Web”  and  a  means  to  easily  navigate 
through  it,  via  links  to  more  detailed  information.  The  design  features  of  the  summary  page  (see 
Figure  5)  included  a  tripartite  status  code  (Today,  Tomorrow,  and  Long  Range)  supporting  the  need 
expressed  by  users  to  know  “how  are  we  doing  now,  and  how  are  we  going  to  be  doing  in  the 
future?”  A  consistent  format  was  maintained  across  the  summary  pages  from  the  different  functional 
areas  supporting  rapid  integration  across  them. 

Several  issues  were  identified,  however,  regarding  the  summary  pages  and  the  information  that 
they  linked  to.  These  needs  must  be  addressed  in  the  future  design  of  summary  pages.  These  include: 
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•  Visibility  issues:  Layout  and  format  changes 

•  Content  and  tools 

A  proposed  change  in  layout  and  configuration  of  future  Knowledge  Walls  (discussed  above)  fea¬ 
turing  fewer  and  larger  displays  would  alleviate  many  of  the  visibility  issues  related  to  the  summary 
pages  themselves.  However,  the  information  on  many  of  the  documents  that  the  summary  pages  pro¬ 
vided  links  to  were  perceived  as  difficult  to  see,  even  when  they  were  brought  into  focus  on  the  large 
focus  monitors.  Business  rules  that  specify  minimum  sizes  for  text  characters  and  objects  should  be 
implemented  to  ensure  that  the  information  on  linked  documents  can  be  read  easily  by  users. 

The  summary  pages  were  designed  to  provide  users  a  means  to  rapidly  acquire  and  integrate 
information  by  providing  them  with  a  high-level,  consistent  summary  from  the  various  functional 
areas.  Although  K-Wall  users  seemed  to  be  satisfied  with  the  general  categories  of  information  dis¬ 
played  on  the  summary  pages — current,  short  term  and  long  term  status  and  alerts,  impacts,  links  and 
graphics  related  to  alerts — several  new  content  needs  did  emerge  from  the  data.  One  of  these  was  the 
need  for  the  content  to  be  tailorable,  to  meet  the  information  requirements  of  different  K-Wall  users. 
Specific  content  needs,  common  to  all  K-Wall  users,  were: 

•  The  age  of  information  presented  on  the  summary  pages  and  links.  The  current  K-Wall 
implementation  provided  time  stamps  on  the  summary  pages  themselves,  indicating  the  most 
current  update  time.  The  links,  however,  did  not  usually  include  any  indication  of  information 
age.  An  intuitive  and,  preferably,  graphical  means  of  indicating  the  age  of  information  that  is 
consistently  displayed  at  all  levels  of  drill-down  should  be  included  in  future  summary  page 
designs. 

•  The  source  of  documents  and  other  links.  When  K-Wall  users  access  a  link,  they  need  to 
remember  the  path  that  they  took  to  access  it,  e.g.,  which  summary  page  the  current 
information  was  linked  from.  Indicating  this  type  of  information  supports  ease  of  navigation 
and  prevents  users  from  “getting  lost”  in  the  drill-down  hierarchy.  The  most  obvious  way  to 
implement  this  would  be  to  provide  the  path  of  the  link,  in  the  same  way  that  Web-browsers 
currently  present  source  information.  However,  a  better  solution  might  be  to  present  a  more 
graphical  representation,  such  as  a  tree  diagram,  that  indicates  to  the  user  what  the  information 
they  are  viewing  was  linked  from,  what  other  paths  could  lead  to  the  current  information  and 
the  relationship  between  information  at  the  same  level  of  drill-down  (see  Card,  Mackinlay  & 
Shneiderman,  1999). 

•  Tools  that  alert  users  to  important  changes  in  summary  pages  and  links.  K-Wall  users  need  to 
be  informed  when  there  is  an  important  change  on  one  of  the  summary  pages  or  to  one  of  the 
documents  they  link  to.  This  need  was  indicated  by  the  various  methods  (business  rules, 
monitoring  the  blue  links,  e-mail,  etc.)  adopted  by  K-Wall  users  in  Global  2000  to  become 
apprised  of  changes  in  information.  The  design  of  the  summary  page  must  take  into  account 
how  to  indicate  that  a  change  has  occurred,  what  the  state  of  the  information  previous  to  the 
change  had  been,  and  at  what  level  (of  drill-down)  the  change  occurred.  A  difficult  task  will  be 
to  implement  methods  that  alert  change  in  a  way  that  is  informative  yet  not  overly  distracting 
to  K-Wall  users. 

•  Tools  that  provide  feedback  to  information  providers  about  K-Wall  usage  and  information 
requirements.  In  order  for  information  providers  to  be  alerted  to  the  presence  of  change  in  their 
information  products,  they  must  know  which  of  these  products  have  been  viewed  by  the  K- 
Wall  users.  Future  K-Wall  implementations  need  to  include  a  means  for  the  K-Wall  users  to 
provide  feedback  to  the  information  providers  that  they  have  accessed,  for  example,  a  docu- 
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ment  or  picture  via  a  hyperlink  on  a  summary  page.  This  feedback  tool  might  require  users  to 
explicitly  provide  this  sort  of  feedback  through  the  use  of,  for  example,  check  boxes  or  other 
controls,  or  it  might  be  transparent  to  K-Wall  users  and  automatically  update  information  pro¬ 
viders  every  time  a  link  is  accessed.  Implementation  of  any  feedback  tool  must  also  take  into 
account  how  best  to  represent  this  feedback  to  the  information  providers. 

K-Wall  users  also  need  a  mechanism  to  communicate  to  information  providers  their  informa¬ 
tion  needs  and  to  provide  them  with  their  goals  and  CCIRs.  One  solution  to  this  problem  might 
be  a  tool,  integrated  with  the  K-Wall,  that  allows  K-Wall  users  to  send  this  type  of  information 
to  the  information  providers.  Another  solution  might  be  to  actually  give  the  information  pro¬ 
viders  their  own  window  into  the  K-Wall,  or  their  own  version  of  the  K-Wall,  so  that  they  can 
see  which  type  of  information  the  K-Wall  users  are  attending  to  and  provide  relevant  informa¬ 
tion  based  on  that.  Of  course,  any  window  or  workstation  that  provides  such  a  view  of  what  the 
K-Wall  users  are  viewing  must  be  designed  with  all  of  the  above  issues  in  mind. 
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5.  CONCLUSIONS  AND  RECOMMENDATIONS 


Observing  and  evaluating  the  use  of  the  K-Wall  prototype  during  a  realistic,  operational  exercise 
provided  invaluable  insights.  Overall,  the  Knowledge  Wall  proved  to  be  a  success.  Support  applica¬ 
tions  such  as  SumMaker  and  TacGraph  were  also  considered  highly  successful.  Most  importantly, 
the  concept  of  a  “Knowledge  Web,”  and  tools  such  as  the  Knowledge  Wall  and  Summary  Pages  that 
afforded  a  view  into  this  web,  were  proven  superior  to  existing  tools  and  processes.  The  data  and 
observations  collected  during  the  Global  2000  War  Game  (reported  here  and  elsewhere)  point  to 
specific  needs  that  must  be  met  by  future  Command  21  research  and  development  efforts.  We  recom¬ 
mend  future  K-Wall  designs  be  developed  and  evaluated  to  address  the  needs  set  forth  in  this  report. 
In  particular,  the  following  needs  should  be  addressed: 

•  Cognitive  tools  to  support  attention  management  and  change  detection,  including  tools 
facilitating  navigation  to  changed  pages  in  the  Knowledge  Web. 

•  Tools  and  improved  business  processes  to  support  multi-tiered  collaboration,  including  feed¬ 
back  and  guidance  for  information  providers  on  content  access  of  their  pages. 

•  Design  layouts  to  support  improved  text  legibility. 

•  Information  integration  across  functional  areas — and  to  the  extent  possible,  this  integration 
should  be  graphical  in  nature. 

•  Information  age  and  source  information  on  the  summary  pages. 
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APPENDIX  A:  DATA  COLLECTION  SUMMARY 


Data  were  collected  by: 

1 .  Automated  data  collection:  Access  patterns  and  K-Wall  configuration. 

a.  Logs  of  URLs  (timestamp,  window,  URL). 

b.  Keystroke  data:  focus,  hide/show  refresh,  presets. 

c.  Screen  captures:  every  5  minutes. 

2.  Time  sampling:  of  usage,  configuration,  verbalizations,  preference  etc. 

3.  Critical  event/situation  summary.  [Provides  context  over  time.] 

4.  Ergonomic  analysis. 


DVs  reslated  to: 

Source 

Data 

Usage  patterns 

Time  Sampling 

Observed  K-Wall  configuration 
Observed  K-Wall  usage 
Verbalizations 

Nonverbal  behaviors 

Log  Files 

URLs  visited  (Time  and  Window) 
Key  presses  (focus,  show/hide) 

Screen  Captures 

Pictures  of  K-Wall  content  and 
configuration 

Usability,  interaction  with 
K-Wall 

Time  Sampling 

Verbalizations 

Nonverbal  behaviors 

Observed  K-Wall  usage 

New  Requirements 

Time  Sampling 

Requests  for  information, 
suggestions,  comments, 
complaints 

Preferences 

Time  Sampling 

Answers  to  preference  questions 
Suggestions,  comments, 
complaints 

Visibility 

Time  Sampling 

Answers  to  visibility/legibility 
questions 

Ergonomic  Analysis/ 
Measurement? 

Viewing  distance/legibility 

Axis  of  visibility 

Glare 

Occlusion 

Ergonomic  issues  related 
to  K-Wall  use 

Ergonomic  Analysis 

Room  layout  issues 

Environmental 
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APPENDIX  B: 

AUTOMATED  DATA  COLLECTED  DURING  GLOBAL  2000 


The  tables  correspond  to  the  figures  presented  in  the  report  text. 

Table  1 .  The  percentage  of  time  that  tactical  information  was  displayed  in  one  or  both  of  the  K-Wall 
focus  displays  (Monitors  9  and  10)  as  a  function  of  format  of  information  and  operational  phase. 


Tactical  Focus 

C2PC  in  both  Focus  monitors 

C2PC  in  one  Focus  monitor 

TacGraph  in  both  Focus  monitors 

TacGraph  in  one  Focus  monitor 

Tables  2-3.  Top  table,  percentage  of  time  that  a  summary  page  was  displayed  on  the  K-Wall 
displays,  averaged  across  all  monitors.  Bottom  table,  percentage  of  time  that  a  summary  page  from 
one  of  the  13  Functional  Areas  was  displayed  on  the  K-Wall,  averaged  across  all  monitors.  [SP  = 
Summary  page;  AD  =  Air  Defense,  DS  =  Deep  Strike,  Eff  =  Effects,  FC  =  Fires  Coordination,  GCC  = 
Ground  Control/CAS,  ISR  =  Intelligence  Sensors  Recognizance,  IW  =  Information  Warfare,  LOG  = 
Logistics,  OMFTS  =  . . .  From  the  Sea,  SOF  =  Special  Operations  Forces,  SC  =  Sea  Control,  RS  = 
Rear  Security,  TMD  =  Theater  Missile  Defense]. 


File  Formats — All  Monitors 

SPs 

C2PC 

Other 

File  Formats — All  Monitors 

AD 

DS 

EFF 

FC 

GCC 

ISR 

IW 

LOG 

OMFTS 
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SOF 

SC 

RS 


TMD 

Table  4.  The  percent  of  time  that  output  of  the  few  cognitive  tools  was  displayed  on  one  the  K-Wall 
focus  monitors. 


Tools  Displayed  on  K-Wall — Focus  Monitors 

C2PC 

WIGS 

TAPS 

TAG 

JFLEX 

CAESAR 

Table  5.  The  number  summary  pages  and  other  information  products  (i.e.,  links)  from  the  13 
functional  areas  that  were  accessed  via  the  K-Wall,  as  a  function  of  operational  phase. 


Number  of  Information  Products  from  the  1 3  Functional  Areas  Accessed 

Summary  Pages 

Links 

Table  6.  The  number  of  links  accessed  per  hour  of  game  play  via  the  K-Wall  as  a  function  of  format 
and  day  of  game  and  operational  phase. 


Number  and  Format  of  K-Wall  Pages  Accessed  as  a  Function  of  0| 

Derational  Phase 

HTML 

MSWord 

MSPowerPoint 

MS  Excel 

WIGS 
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Tables  7-8.  The  number  of  functional  area  summary  pages  and  information  products  accessed  via 
the  K-Wall  as  a  function  of  operational  phase  of  the  game.  [AD  =  Air  Defense,  DS  =  Deep  Strike,  Eff 
=  Effects,  FC  =  Fires  Coordination,  GCC  =  Ground  Control/CAS,  ISR  =  Intelligence  Sensors 
Recognizance,  IW  =  Information  Warfare,  LOG  =  Logistics,  OMFTS  =  . . .  From  the  Sea,  SOF  = 
Special  Operations  Forces,  SC  =  Sea  Control,  RS  =  Rear  Security,  TMD  =  Theater  Missile  Defense, 
OVV  =  Overview,  BC  =  Blue  Comms,  CSR  =  CAESAR] 


Number  of  Summary  Pages  Accessed  as  a  Function  of  Operational  Phase 

AD 

DS 

EFF 

FC 

GCC 

IW 

ISR 

LOG 

OMFTS 

RS 

SC 

SOF 

TMD 

OVV 

TAPS 

BC 

CSR 

CJTF 

Number  of  Other  Information  Products  Accessed  as  a  Function  of  Operational  Phase 

AD 

DS 

EFF 

FC 

GCC 

IW 

ISR 
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LOG 

OMFTS 

RS 

SC 

SOF 

TMD 

OVV 

TAPS 

BC 

CSR 

CJTF 

Table  9.  The  percentage  of  critical  events  displayed  on  the  K-Wall  as  a  function  of  the  format  source 
of  the  information  and  the  criticality  of  the  event  (1  =  critical  event,  2  =  significant  event,  3  = 
important  event,  4  =  event  of  interest). 


Percentage  of  Critical  Events  Indicated  on  K-Wall  as  a  Function  of  Source  and  Criticality 

Criticality  1 

Criticality  2 

Criticality  3 

Criticality  4 

Table  10.  The  average  amount  of  time  it  took  for  critical  events  to  be  displayed  on  the  K-Wall  as  a 
function  of  the  format  source  of  the  information  and  the  criticality  of  the  event  (1  =  critical  event,  2  = 
significant  event,  3  =  important  event,  4  =  event  of  interest). 


Percentage  of  Critical  Events  Indicated  on  K-Wall  as  a  Function  of  Source  and  Criticality 

Criticality  1 

Criticality  2 

Criticality  3 

Criticality  4 
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Table  11.  The  K-Wail  operations  performed. 

Number  and  Type  of  K-Wall  Operations  Accessed  as  a  Function  of  Operational  Phase 


Preset  1 

Preset  2 

Preset  3 

Preset  4 

Hide 

Show 

Refresh 

Focus9 

Focusi 0 

Home 
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APPENDIX  C: 

OBSERVATIONAL  DATA  COLLECTED  DURING  GLOBAL  2000 


Note:  Observations  made  are  presented  in  italicized  text.  Many  of  them  are  paraphrased  or  have 
been  corrected  so  that  they  are  grammatically  correct.  Comments  taken  from  documents  (such  as  the 
KM’ s  documents)  are  presented  in  italicized,  underlined  text.  Comments  made  during  Global  2000 
and  Executive  sessions  are  displayed  in  normal  font. 

OBSERVATIONS  AND  PLAYER  COMMENTS 

Information  Requirements 

Did  the  interviews  identify  the  right  information  requirements? 

Do  the  K-Wall  design  solutions  meet  these  requirements? 

Are  there  new  K-Wall  requirements? 

14  REQUIREMENTS  IDENTIFIED  IN  INTERVIEWS 

General  Requirements 

1 .  (Shared?)  SA 

•  Just  looking  at  this  thing  [the  K-Wall],  it’s  not  intuitively  obvious  what  we’re  doing. 

•  It  was  useful  for  the  CJTF;  it  was  the  only  tool  available  .  .  .  normally  he  would  get  SA  from 
radios,  people  talking  to  each  other  . . .  here  the  K-Wall  was  the  only  way  to  get  SA. 

•  We  need  a  common  picture  that  gives  instant  SA  to  the  commander  when  they  walk  into  the 
room  . .  .  this  is  a  pretty  interesting  first  step. 

•  Life  would  have  been  miserable  if  I  didn’t  have  this  . . .  because  I  would  have  been  constantly 
pinging  the  other  guys  about  what’s  going  on. 

•  The  CJTF  did  not  have  to  2nd  guess  commanders  because  of  comfort  he  had.  .  . . 

•  SA  increased  in  the  CJTF  but  not  component  commands,  partly  because  of  the  inability  to  put 
together  a  plan,  intent. 

•  We  didn’t  do  a  briefing  at  8  o’clock,  1  o’clock  ...  we  didn’t  need  to  be  brought  up  to  date 
because  we  were  always  up  to  date. . . . 

•  It  was  useful  for  the  CJTF  ...  we  had  shared  information,  distributed  sources. . .  . 

•  ...  I  was  very  comfortable  that  I  was  getting  the  information  I  needed. . . . 

•  ...  The  K-Wall  prevented  information  flow.  .  .  . 

•  Speed  of  Command  was  increased  by  the  improved  SA  of  the  CJTF.  The  K-Wall  was  a  factor 
in  terms  of  improved  speed  of  command.  SA  was  not  good  in  the  functional  cells. 

•  [Regarding  “working  info  into  knowledge’’]  When  I  walked  around  during  game  play,  this  [the 
K-Wall]  was  the  only  place  it  was  happening. 

•  The  JFACC  is  holding  a  “live”  situation  brief  every  morning,  efficient  and  direct,  which  is  not 
possible  using  IT  tools  alone. 

•  The  K-Wall  is  valuable  in  giving  all  a  sense  of  what’s  going  on,  especially  the  CJTF  and  the 
CINC. 

•  The  K-Wall  does  a  reasonable  job  of  keeping  awareness  level  to  CJTF  and  his  staff. 

•  I  found  the  K-Wall  useful.  The  CJTF  depends  on  the  K-Wall  almost  entirely  for  awareness. .  . . 
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•  Yes,  it’s  a  K-Wall,  it  is  a  wall  that  prevents  any  knowledge  from  reaching  the  CJTF. 

•  It  captures  ideas  that  the  K-Wall  is  actually  supporting  and  continues  knowledge  transfer  vice 
periodic  knowledge  transfer.  It  helped  free  players  from  the  tyranny  of  preparing  for  a  daily 
brief. 

•  I  saw  several  cycles  of  support  when  the  CJTF,  prior  to  [Phase  2],  had  time  to  monitor  the 
board — the  functional  commanders  were  making  a  great  effort  to  keep  the  boss  informed.  As 
the  pace  of  activity  increased  due  to  the  transition  to  [Phase  2],  they  reverted  to  other  means  of 
maintaining  awareness,  primarily  physical  meetings  and  email,  and  the  CJTF  became 
overwhelmed  and  was  unable  to  focus  on  the  bigger  picture  on  the  wall. 

•  It  allowed  the  CDRs  to  rapidly  assess  information  for  use  in  making  decisions. 

•  The  value  grew  over  time  as  both  an  SA  and  synchronization  tool  as  separators  became 
familiar  with  both  the  concept  and  how  to  feed  into  it. 

2.  Integration 

•  Only  the  CJTF  can  look  at  all  the  screens  at  the  same  time. 

•  I  see  this  as  valuable  because  we  can  put  different  formats  up  there  to  discuss  things. 

•  The  K-Wall  was  a  good  first  step.  We  need  better  integration  ...  it  was  hard  to  see 
everything.  .  .  . 

•  The  CJTF  is  the  integrator  because  of  more  screens. 

•  An  advantage  we  had  over  the  component  commanders  was  not  having  a  single  page  but 
multiple  screens. 

•  It’s  hard  to  integrate  stoplights  [status  alerts]. . . . 

•  It  allowed  us  to  look  at  the  same  information  at  the  same  time  and  discuss  it. 

•  This  is  the  only  place  where  you  can  bring  everything  together,  [regarding  the  NCA  having 
access  to  summary  pages  but  not  the  K-Wall] 

•  The  information  presented  on  the  wall  was  not  fused  from  relevant  functional  areas.  However, 
it  gave  the  false  impression  that  it  was.  .  . . 

•  Fewer  screens  with  integrated  data  would  be  a  better  answer. 

•  The  next  step  is  ...  a  relationship  between  the  FA  automatically  updating  .  .  .  and  fusing 
information  .  .  .  tailored  to  a  job  or  function.  .  .  . 

Format  Requirements  [NB  these  comments  are  related  to  general  K-Wall  format  only,  specific 
format  of  summary  pages  are  below] 

3.  Intuitive,  Graphicai 

•  The  technology,  a  lot  of  it  being  the  K-Wall,  was  fairly  intuitive.  .  .  . 

4.  Consistent 

•  An  observed  problem  is  that  navy  personnel  don’t  know  what  army  symbols  are — a  suggestion 
was  made  that  there  should  be  a  roll-over  function  in  TacGraph  that  shows  a  description  of  a 
unit. 

•  There  is  no  way  to  have  one  metric  for  all  FAs  ...  If  we  were  together  for  a  longer  time,  a 
metric  would  develop. 

•  We  need  standards  of  use  (e.g.,  colors  used,  processes  completed,  level  of  detail,  purposes, 
etc.). . . . 
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•  We  need  to  determine  what  level  of  information  you  see  at  first  blush. 

Content  Requirements 

“Content  and  people  make  the  K-Wall  useful — without  them,  it  is  simply  monitors.” 
“Garbage  in-garbage  out.” 

5.  Tactical  focus 


6.  Supplemental  information 

(Other  content  provided — Overview  page,  etc.) 

•  When  I’m  looking  for  pages,  I  don’t  use  the  overview  slide. .  . . 

•  At  the  beginning  of  Phase  2,  the  Overview  page  was  no  longer  being  displayed,  and  no  one 
seemed  to  use  it . .  .  later  in  Phase  3,  the  Overview  page  was  brought  back  up.  [NB  is  different 
from  the  K-Wall  operator] 

•  The  K-Wall-Op  reported  that  he  used  the  red  alert  on  the  Overview  page  to  know  to  look  at  the 
hidden  Rear  Security  page. 

7.  Goals  and  CCIRs 

•  It  did  not  provide  the  commanders  intent  to  me  . . .  how  come  I  had  to  come  up  here  to  tell  the 
commander  what’s  going  on. 

•  The  CJTF  level  did  not  feel  comfortable  . . .  with  no  way  to  sync  with  higher  and  lower 
levels — no  way  to  push  info  or  convey  needs/plans/intentions. 

•  [Discussing  the  CJTF  page]  We  need  to  include  things  like  the  commander’s  intent.  Today  ' 
(what  we  plan  to  do  for  next  10  days). . . . 

•  We  need  to  restrict  what  goes  on  the  K-Wall.  It  should  be  top-down  drive. . . .  The  CINC  gives 
CCIRS. . . .  but  these  are  mission  dependent. . . .  We  really  need  a  constant  baseline  for 
information  and  mission  related  additions. 

•  When  the  component  commanders  let  the  CJTF  know  they  were  having  trouble  knowing  what 
the  CJTF  wanted,  it  led  to  the  development  of  the  a  CJTF  page. 

•  Need  to  define  better  what  “Today,”  “Tomorrow,”  and  “Long  Range”  mean.  Some  areas  (e.g.. 
Sea  Control)  used  them  better  than  others.  Information  requirements  and  CCIRs  should  be 
considered. 

•  Commander’s  priorities  must  be  made  clear  to  all.  Shared  awareness  must  include 
CCIRS/priorities  etc.,  or  it’s  not  shared. 

•  The  overall  battle  plan  is  not  really  reflected  on  the  K-Wall. .  . . 

8.  Anchor  Desk  Output 

•  FA  summaries  on  the  K-Wall  100%  of  the  time 

•  On  focus  monitors  X  %  of  the  time 

9.  Connectivity/Collaboration  (K-Wall,  IWS,  VTC) 
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•  One  problem  is  that  when  [a  problem]  is  supposed  to  be  taken  care  of  by  Navfor,  how  do 
I  know  whether  they’ve  taken  care  of  it?  ...  Do  I  need  to  send  them  an  e-mail  asking 
about  it?  ...  . 

•  The  K-Wall  is  really  a  centralized  control  at  the  top,  not  a  component.  How  do  we  make 
collaboration  work?  Components  need  to  collaborate. 

•  Groups  that  were  on  IWS  did  use  chat  for  collaboration. 

•  There  was  a  need  for  better  shared  awareness/collaboration. 

•  The  VTC  was  worthless,  actually  negative  because  poor  execution  made  it  a  distraction  . . . 
maybe  they  thought  they  were  sharing  information  but  they  weren’t.  It  should  only  be  used  in 
certain  situations. 

•  What  made  it  work  was  the  table,  people  around  the  table,  verbal  communication  . . .  great  for 
passing  information  up  and  down  . . .  but  I  missed  face  to  face  and  verbal  communication  from 
commanders  to  work  . . .  needed  to  get  the  headphones  to  work,  the  VTC  to  work. 

10.  Cognitive  toois  (see  new  tools  and  SP  tools) 

Feature  Requirements 

11.  Flexible  configuration 

Across  Contexts 

•  Now  that  we  are  no  longer  in  [Phase  1]  and  in  [Phase  2],  I  need  to  see  information  relevant  to 
that . . .  only  certain  pages. 

•  At  the  beginning  of  Phase  2,  the  Overview  page  was  no  longer  being  displayed,  and  no  one 
seemed  to  use  it . .  .  later  in  Phase  3,  the  Overview  page  was  brought  back  up.  [NB  is  different 
from  the  K-Wall  operator] 

•  We  need  to  replace  that  overview  with  ground  control. 

•  The  limitations  were  lack  of  flexibility  in  reconfiguring  it  and  the  display  technology  inability 
to  export  pictures  to  show  other  locations  in  the  JCC  and  components  that  the  CDR  was 
focusing  his  attention  upon  them. 

Across  Users 

•  Whose  tool  is  it?  If  three  groups,  CJTF,  CINC,  NCA,  can  see  the  K-Wall  they  need  to  see 
different  information  .  .  .  One  K-Wall  glove  does  not  fit  all  -  there  needs  to  be  tailorable 
information.  .  .  . 

•  The  big  issue  is  who  uses  it.  Information  presented  should  differ.  It  should  be  tailored  to 
specific  users/user  groups. 

12.  Drill  down  (also  see  frequency,  depth,  content  of  drill-down  in  usage  logs,  screen  captures) 

•  There  is  a  problem  with  too  many  links,  I  don’t  want  to  drill  down  to  say  “mother  may  I.” 

•  When  I  click  on  a  red  gumball  there  are  no  links  .  .  .  drill-down  is  not  being  supported. 

•  When  you  hit  a  button,  (today,  tomorrow,  long  range),  there  should  be  an  indication  of  what’s 
going  on. 

•  I  just  want  our  guys  [ISR  ADs]  to  be  thinking  about  how  to  update  information,  so  you  can 
drill  down. 

•  Some  of  the  dots  have  changed  color  but  they  still  don’t  have  links  associated  with  them. 

•  An  example  of  what  people  called  a  “good  link”  was  the  METOC  brief  using  the  K-Wall — the 
grids  have  links  to  drill  down,  color  coding  etc. 
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[In  a  comment  about  how  intuitive  technology  was^  We  could  view  the  FA  information,  could 
drill  down  for  more  information. 


Information  Source 

•  The  problem  is  that  pages/links  don’t  indicate  where  they  came  from  . . .  need  to  include 
addresses  or  headings  on  links. 

•  Left  side  [information  in  focus  monitor  9]  is  from  . . .  ? 

13.  information  Age  and  Reliability 

•  One  thing  we’ve  heard  about  the  K-Wall  this  week  is  “how  current  is  the  information?” 

•  [Pointing  out  how  long  it  takes  to  update  the  page,  and  how  this  leads  to  outdated  information 
being  displayed  on  the  K-Wall]  The  problem  is  that  it’s  not  plugged  into  anything  ...  it  takes 
so  long  to  update,  throw  on  overlays,  that  it’s  old  by  the  time  it  gets  up  there. 

•  A  lot  of  these  things  don’t  have  times  on  them,  so  we  don’t  know  how  old  they  are. 

•  It’s  been  good  when  the  guys  are  updating  it. 

•  Currency  of  information  is  a  big  issue  with  the  wall,  in  terms  of: 

-  how  long  to  produce  products 

-  how  to  present/link/publish  products 

14.  Tactical  overlays 

•  It  would  be  very  useful  to  have  access  to  a  map/chart  [like  a  NIMA  map]  that  we  can  put  on 
the  K-Wall . .  .  This  is  a  whole  new  category  to  go  on  the  K-Wall:  access  to  a  “reference 
library”  of  our  own.  Things  on  WIGS,  order  of  battle,  capabilities  of  ships,  reference  manuals, 
maps  . .  .  with  a  searchable  database  . . .  everything  that  I  have  on  notebooks  right  now. 

•  We  need  a  tool  for  the  wall  with  overlays,  to  display  METOC  and  ISR. . . . 

•  There  is  lots  of  use  of  TacGraph  tactical  pictures  on  sununary  pages,  especially  2nd  level  links. 

•  There  is  the  use  of  C2PC  overlays  (see  screen  captures). 

•  In  a  perfect  world  you  want  a  real  time  map  that  you  can  add  layers  to  from  METOC,  sensors 
etc.  ...  a  pull  vs.  a  push  system.  .  .  . 

•  There  is  the  need  for  layered  data  controlled  by  the  DMs  (i.e.,  overlays) — fused,  configur¬ 
able.  . . . 

•  Players  are  actually  trying  to  find  the  full  capability  of  the  systems.  They  are  trying  to  import 
data  from  C2PC  onto  TacGraph.  Both  systems  have  good  features  that  the  other  doesn’t.  They 
want  real  time  data  from  C2PC  on  the  TacGraph  chart  for  manipulation. 

•  .  .  .  most  of  us  desire  a  consistent  tactical  picture.  It’s  mainly  semantics,  but  I’d  arsue  that  the 
C2PC  is  inappropriate  for  the  CJTF  and  above.  But  those  procedures  for  providing  the 
“picture”  need  refinins.  Clearly  tools  like  TacGraph  are  ereat  for  rapidly  building  a  picture 
that  can  express  information  concisely  and  rapidly  .  .  .  need  trainins,  not  on  how  to  operate  the 
tool  but  on  how  information  can  be  presented .  .  .  visualizine  information. . . . 
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NEW  K-WALL  REQUIREMENTS 


NEW  CONTENT 
Information  Source 

•  The  problem  is  that  pages/links  don’t  indicate  where  they  came  from  .  . .  need  to  include 
addresses  or  headings  on  links. 

•  Left  side  [information  in  focus  monitor  9]  is  from  .  .  .  ? 

Other  (non-FA)  Content 

•  Requests  for  or  development  of  pages  with  information  from: 

-  CJTF  (to  push  to  component  commanders,  up  to  CINC) 

-  METOC 

-  Blue’s  Red  cell 

-  Blue  comms 

-  Searchable  reference  library — maps,  ROEs,  capabilities,  etc. 

New  Tools 

Alerting  Tools  (see  SP  tools  but  idea  also  applies  generally  to  K-Wall) 

•  There  have  been  lots  of  requests  for  the  CJTF  to  have  control  of  alerts — to  be  able  to  alert 
everyone  else  (i.e.,  push  information  to  everyone  else). 

•  The  pages  that  aren’t  displayed  on  the  K-Wall  don’t  get  looked  at. 

Attention  Management  (pointers  etc) 

•  [To  collaborate...]  We  each  need  to  have  our  own  different  colored  pointers  so  we  can  talk 
about  the  same  things  together. 

•  Briefings  and  discussions  using  the  K-Wall  frequently  involved  the  use  of  a  laser  pointer. 

•  You  need  the  ability  to  have  a  pen  or  light  you  can  use  to  draw  on  the  board  from  a 
distance  ...  a  mouse  pointer/pencil  device. 

Feedback  to  ADs 

•  We  need  a  way  for  K-Wall  users  to  convey  desires/disseminate  information  “down”  to  anchor 
desks  and  lower  echelons. 

•  [AD  in  the  same  room  as  the  JCC]  I  have  an  advantage  over  people  downstairs  because  I  can 
walk  over  and  see  if  they’ve  looked  at  my  links  or  not  [using  the  color  of  the  link  text  as  an 
indicator], 

•  Give  the  admirals  a  “laser”  and  check  boxes  that  they  can  check  once  they’ve  read  something. 

•  ...  there  should  be  a  feedback  loop  and  rules  for  stoplights. 

•  There  needs  to  be  automatic  feedback  that  the  page  has  been  read. 

•  A  KM  fix  (business  rule)  was  to  add  an  asterisk  a  new  link  and  have  the  BWC  contact  the  AD 
when  it’s  been  read  (via  e-mail).  Also,  include  an  exclamation  point  after  a  link  that  the  AD 
thinks  really  needs  to  be  read/addressed. 

•  There  needs  to  be  a  feedback  loop  on  the  K-Wall  from  the  BWC  to  the  AD/FA. 
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Feedback  notification  is  required  when  something  urgent  has  been  received  by  the  CJTF. 
I  recommend  that  the  signal  be  cleared  once  the  CDR  views  the  status. 


Other 

•  Built-in  chat  with  functional  areas  for  summary  pages  would  be  useful. 

•  Webcam32  might  be  a  tool  for  future  use  on  the  wall. 

New  Methods  of  K-Wall  Operation/Access 

•  It  would  be  nice  if  the  K-Wall  was  connected  to  the  BW  displays,  so  when  the  BWCs  say  we 
need  to  look  at  something,  there’s  a  button  that  brings  up  the  content  of  their  display  onto  the 
wall. 

•  There’s  an  issue  of  putting  old  files  in  an  archive  that  can  be  liked  to  through  the  wall. 


K-WALL  USAGE  AND  USABILITY 
Usage  Context:  When  and  How 

•  The  K-Wall  was  used  as  a  “situation  assessment  tool,”  continuously  viewed  by  JCC  staff. 

•  The  K-Wall  was  used  as  briefing  tool  by  component  commanders  to  individual  or  groups  of 
JCC  staff 

•  They  (AD  personnel)  have  acknowledged  that  the  K-Wall  is  more  important  to  briefings. 
Everything  that’s  there  can  be  used  to  brief .  .  .  Everyone  needs  to  be  updating  their  pages. 

•  ...  a  good  tool  to  transition  in  and  out  of  briefing  and  for  [assessment]. 

•  The  K-Wall  was  used  to  review  information  to  put  on  the  CJTF  page. 

•  The  K-Wall  was  used  for  briefing,  discussion  of  information  without  having  to  stop  the  war. 

•  We  didn’t  do  a  briefing  at  8  o’clock,  1  o’clock  . . .  we  didn’t  need  to  be  brought  up  to  date 
because  we  were  always  up  to  date  . .  .  we  didn’t  have  to  go  off  for  2  hours  and  prepare  a  brief. 

Usability  Problems,  Ease  of  Use 

•  It’s  an  issue  (training  or  otherwise)  that  the  information  providers  need  to  not  put  a  link  to  a 
document  that  they  are  working  on.  The  K-Wall  operator  needs  to  know  what’s  open 

•  When  2  [identical]  pages  are  on  the  K-Wall  (e.g.,  on  a  peripheral  and  a  focus  monitor)  and  a 
link  is  clicked  on  one  of  them,  the  color  of  the  link  only  changes  on  that  page  and  not  the  other 
one.  You  need  to  show  the  change  on  both  pages. 

•  The  pages  that  aren’t  displayed  on  the  K-Wall  don’t  get  looked  at. 

•  Where’s  my  mouse?  [wiggling  the  mouse  to  find  the  cursor] 

•  It  was  found  to  be  useful  by  most  users.  .  .  . 

•  Navigating  using  the  channel  menu  is  inconvenient.  For  example,  getting  to  WIGS  involves 
many  steps. 

•  With  less  than  2  weeks  of  use,  the  team  was  able  to  efficiently  use  the  K-Wall. 

•  The  K-Wall  is  useful  but  it’s  not  easy  to  move  contents  to  another  screen.  Example:  moving 
the  ISR  picture  to  the  bis  screen. 
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Visibility  of  K-Wall 

•  If  you’re  near-sighted,  you  can  look  through  fingers  [holding  fingers  to  eye]  to  see  it. 

•  When  I  look  at  the  picture  on  my  page,  I  can’t  see  it.  [not  a  problem  once  it’s  clicked  and 
enlarged]. 

•  It’s  hard  to  see  items  in  pull  down  menus  when  trying  to  bring  up  a  new  summary  page  [in 
monitor  11]. 

•  Here’s  another  part  of  the  K-Wall:  you’ve  got  to  have  good  eyes. 

•  The  K-Wall  operator  frequently  walked  up  to  monitors  9  and  10  to  read  them. 

•  K-Wall  users  occasionally  printed  pictures  to  look  at  them  (more  common  on  the  Coronado). 

•  He  [the  CJTF]  couldn’t  see  the  Strike  page,  so  he  brought  it  up  on  his  laptop. 

•  Whenever  I  say,  “hey  COS  look  at  this  on  the  wall,”  he  has  to  get  out  his  glasses. 

•  Can  you  see  that  tab?  [on  bottom  of  Excel  page  on  M9 — other  BWC’s  head  blocking  his 
view]. 

•  With  all  the  monitors,  needing  to  focus  eyes  is  a  problem.  If  people  have  a  headache  they  can’t 
make  decisions. 

•  Many  of  the  monitors  are  too  far  away  to  see  easily  in  JCC. 

Who  used/viewed/attended  the  K-Wall? 

•  Some  personnel  (e.g.,  J3)  primarily  looked  at  a  laptop,  but  occasionally  did  use  the  K-Wall  to 
discuss  information  with  others.  Later  these  personnel  used  the  K-Wall  much  more.  Others 
(CJTF,  COS,  J6,  etc)  looked  at  the  K-Wall  throughout. 

K-Wall  operation:  who  operated  the  K-Wall,  who  controlled  the  content  of  the  K-Wall, 

methods  used  by  operator/controller? 

Evolution  of  K-Wall  Operation/Control 

•  Early  in  the  game  the  BWC  was  the  operator  of  the  K-Wall  (he  had  experience  from  NDIA) — 
he  had  the  responsibility  of  looking  for  changes,  bringing  them  up  on  the  wall,  and  drawing  the 
attention  of  the  CJTF  staff  to  important  changes/info.  Later,  other  BW  personnel  were  the  K- 
Wall  operators;  and  the  CJTF  personnel  became  more  active,  requesting  that  information  be 
brought  up  on  the  wall  or  onto  focus  screens  to  better  notice  changes. 

•  [CJTF  to  K-Wall  operator,  BWC]  You  should  notice  when  changes  occur  and  bring  it  up  on 
the  wall. 

•  I’m  looking  for  blue  text  [=  new  link,  unaccessed]  as  an  indication  of  new  information,  since 
there  isn’t  anything  flashing  to  let  me  know  it’s  changed.  Sometimes  they  [CJTF  et  al.]  ask  me 
to  bring  stuff  up  on  the  wall. 

•  There  was  an  evolution  from  them  looking  for  information  to  bring  up  on  the  wall  to  seniors 
requesting  information  that  they  knew  was  there. 

•  Knowing  what  and  where  information  is  located  (and  that  it  exists)  is  vital  to  making  the  wall 
useful. 

•  The  K-Wall  operator  evolved  into  the  Knowledge  Warrior. .  . . 

•  In  order  for  the  K-Wall  to  be  successful,  there  must  be  a  verson  dedicated  to  overatins  it,  and 
components  must  be  notified  well  in  advance  of  requirements. 


C-8 


Preference  and  Comparison  to  Other  Tools 

•  This  is  much  better  than  what  we  had  before,  better  than  a  stickpin  in  a  map. . . . 

•  The  K-Wall  was  a  hit. . . . 

•  It’s  a  great  information  filter,  but  there  are  problems. . . . 

•  [Issue  of  defining  what  alerts  mean]  wouldn’t  be  important  if  they  didn’t  like  the  K-Wall,  but 
that’s  what  they’re  using. 

•  [Discussing  Global  2000  in  general]  We’re  just  light  years  ahead  of  last  year  .  . .  but  each  year 
should  get  better. 

•  [The  K-Wall]  is  a  good  idea  but  it  needs  some  tweaking. . . . 

•  CINC  had  more  negative  comments. . . . 

•  [From  AD  perspective] ...  as  people  got  used  to  the  tools,  then  there  was  more  use,  more 
“let’s  put  this  on  the  wall”. . . . 

•  The  K-Wall  was  a  good  first  step. .  .  . 

•  Life  would  have  been  miserable  if  I  didn’t  have  this. . . . 

•  It’s  a  great  tool  you’ve  got  here,  don’t  get  me  wrong. . . . 


SUMMARY  PAGES 

Did  the  summary  pages  have  the  right  content/format? 
How  can  the  summary  pages  be  improved? 


Summary  Page  and  Links  Content 
Summary  Page 

•  An  initial  problem  was:  “what  should  be  on  ... .  page?” 

•  It’s  important  to  communicate  and  have  a  common  understanding  of  what  should  be  on  the 
wall. 

•  We  need  to  define  what  24  hours,  48  hours,  long  term  means. 

•  For  ISR  [summary  page],  instead  of  today,  tomorrow,  and  long  range,  I  want  to  see  air,  land, 
surface,  subsurface  with  links  to  C2PC. 

•  Effects  page — the  alerts  changed  to  Past  72  hours.  Next  72  hours.  Long  range. 

•  [Discussing  CJTF page]  We  need  to  include  things  like  the  commander’s  intent.  Today  (what 
we  plan  to  do  for  next  10  days) . . .  what’s  going  on  today,  tomorrow,  etc. 

•  Something  I  know  the  CINC  is  going  to  want  to  see  is  how  we’re  doing  on  Effects  ...  we 
should  highlight  this  . . .  how  we  have  done  . .  .  that’s  [TAPS]  what  we’ve  done.  .  .  . 
unintended  consequences,  etc. 

•  We  didn’t  ever  define  what  today,  tomorrow,  long  range  was  . . .  Sea  Control  had  a  good 
method  =  Today:  what  I’m  going  to  do  today,  intentions,  etc. 

•  [CJTF  page]  We  noticed  we  weren’t  updating  our  seniors  so  we  developed  a  CJTF  page.  Both 
Admiral  Zelibor  and  I  decided  what  would  go  on  it  via  discussion. 

•  [Regarding  status  alerts]  We  need  to  understand  what  each  rule  set  is  and  how  it  is  being  used 
by  each  cell  and  document  it  for  the  CJTF  and  CINC.  The  rules  can  be  different  for  each  FA  as 
long  as  the  CINC  and  CJTF  know  the  rules. 
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•  The  stoplights  don’t  tell  me  anything  because  I  don’t  know  what  metric  was  being  used  ...  we 
didn’t  really  look  at  them.  All  red  told  me  was  that  the  commander  was  uncomfortable  with 
what  was  going  on. 

•  Do  you  have  any  way  of  indicating  effectiveness? — keep  it  at  high  level. 

•  On  K-Wall,  the  overview  stoplight  graph  does  not  automatically  reflect  the  status  of  the 
various  summary  slides  when  they  are  updated. 

•  That’s  what  we  want  to  see:  what  they’re  planning  to  do  today,  what  are  their  intentions  for 
tomorrow. .  .  . 

Links 

•  When  you  hit  a  button,  (today,  tomorrow,  long  range)  there  should  be  an  indication  of  what’s 
going  on. 

•  When  there’s  a  yellow  or  red  alert,  the  functional  areas  should  tell  you  what  they’re  doing 
about  it. 

•  When  you  change  the  color  of  the  alert,  we  need  to  know  why  the  alert  changed. 

•  That’s  what  we  want  to  see:  what  they’re  planning  to  do  today,  what  are  their  intentions  for 
tomorrow. . . . 

•  [CJTF  page]  .  . .  alerts  should  be  . .  .  what’s  imminent,  ROEs,  things  like  that. 

•  [CJTF  page]  The  CJTF  should  provide  the  CINC  with  a  picture  of  what’s  going  on  so  he  can 
let  others  know.  .  .  .  Take  the  best  of  the  best  from  the  other  summary  pages.  Not  specific 
details  but  the  big  picture.  Keep  an  eye  out  for  pages  that  can  be  used  for  this.  We’re  going  to 
link  to  links. 

•  {Pointing  out  that  someone  linked  to  the  battle  plan  in  WIGS  ]  If  it’ s  in  there  and  we  can  find 
it,  we’ll  use  it. 

•  [The  text  on  a  link  from  a  yellow  status  alert  included  a  description  of  why  the  text/alert  was 
yellow]  They  should  all  do  that! 

•  A  lot  of  these  things  don’t  have  times  on  them  so  we  don’t  know  old  they  are. 

•  It  is  a  problem  that  pages/links  don ’t  indicate  where  they  came  from .  . .  need  to  include 
addresses  or  heading  on  links. 

Summary  Page  and  Links  Format 
Summary  Page 

•  Suggested  changing  the  summary  page  layout  so  that  the  entire  page  is  a  picture  with  a  small 
window  of  alerts/links  in  the  comer. 

•  We  need  standards  of  use  (e.g.,  colors  used,  processes  completed,  level  of  detail,  purposes, 
etc.). 

•  There’s  too  little  room  on  the  summary  page. 

•  What  is  lacking  is  a  means  of  summarizing  information  on  the  K-Wall  and  passing  it  up  [to  the 
CINC].  .  . .  What  are  the  implications,  how  are  things  unfolding.  .  .  . 


Links 

•  When  graphics  link  to  PowerPoint  slides — use  simple  text,  no  logos,  no  pictures  . . .  need 
information  . .  .  use  black  and  white  text .  .  .  the  links  are  just  [too]  full  of  stuff. 

•  ...  too  much  data,  need  simple  PowerPoint  slides. 
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•  I  put  the  most  important  information  on  the  last  slide  (of  10) — they  never  even  looked  at  it . . . 
you  need  to  put  the  most  important  information  on  the  first  slide. 

•  If  they  put  a  lot  of  text  on  it,  I  just  skip  right  over  it  [when  monitoring  the  new  links].  The 
information  needs  to  be  condensed. 

•  Keep  things  simple,  use  big  bullets.  . . . 

•  A  business  rule  should  be:  don’t  use  too  many  acronyms  because  not  everyone  understands 
them. 

•  It  is  a  problem  that  pages/links  don’t  indicate  where  they  came  from  . . .  need  to  include 
addresses  or  headings  on  links. 

•  That  was  a  good  link,  that  was  a  good  picture.  [TacGraph  picture,  link  from  SOF  page] 

•  Italicized  text  on  the  link  is  hard  to  read. 

•  Yellow  text . . .  hard  to  read  on  white  background.  The  Marines  don’t  listen  very  well. 

•  If  a  whole  page  of  text  was  on  the  K-Wall,  we  didn’t  even  read  it. 

•  I  want  to  show  [pictures]  like  this  [TAPS],  not  so  much  bean  counting. . . . 

•  I  know  I  can  highlight  these  [objects  on  TacGraph  picture  on  M9,  i.e.,  link  to  information]  but 
there  are  too  many  things  up  there. 

•  Too  much  detail ...  if  you  had  a  picture  showing  the  same  thing,  that  would  be  great,  [list  in 
Word  doc,  small  font] 

•  The  SOF  link  to  the  .avi  file  (SEAL  video),  42MB,  the  AD  says  this  is  how  they  normally  do 
reporting  need  a  definition  of  what  types  of  files  can  be  linked  to,  how  big  etc. 

•  They  are  not  using  TacGraph  enough,  they  should  because  there’s  not  a  lot  of  space  on  a 
page  ...  a  graphic  would  have  told  a  thousand  words. 

•  Several  FAs  were  linking  summary  status  alerts  to  a  PowerPoint  presentation — it  took  forever 
to  update  . .  .  can’t  updates  be  updated  with  an  html  file.  Also  having  a  standard  way  of 
representing  a  report  would  help  SA  . . .  we  are  seeing  different  color  information  with  no  idea 
what  each  color  means.  .  . .  still  having  to  guess  about  why  an  item  is  red. 

•  A  shift  to  savins  all  files  as  html  was  involved  but  the  K-Wall  designers  didn’t  think  that  is 
always  a  better  thins  to  do.  Template  makers  should  look  at  the  size  of  a  file  alter  savins  to 
make  sure  an  html  file  is  not  larger  that  the  orieinal. 


SP  Update 

•  Some  of  the  dots  have  changed  color  but  they  still  don’t  have  links  associated  with  them. 

•  It  is  also  a  problem  that  alerts  stay  the  same  color. 

•  The  ISR  AD  walks  up  to  let  the  BW2  know  that  his  page  is  updated  [a  result  of  decreased 
refresh  rates]. 

•  Is  the  game  paused?  Why  is  nothing  updating? 

•  The  game  is  progressing  smoothly  ...  the  K-Wall  is  getting  updated  regularly. 

•  As  a  process  for  updating  the  K-Wall  (OMFTS).  the  AD  updates  vases  process  involves  links 
to  vases  which  are  automatically  being  updated  by  cognizant  function. 


SP  Tools 

Alerting  when  SP  had  been  changed  or  updated 

•  Change  the  stop  lights  so  they  flash  when  changed,  like  the  gauges  on  WIGS. 

•  There  are  problems  . . .  with  automatic  alerts  and  things  like  that. . .  . 
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•  When  they  update  their  page  they  need  to  update  their  links  [a  problem  with  new  links  not 
being  blue], 

•  ....  there  isn’t  anything  flashing  to  let  me  know  it’s  changed. 

•  On  the  K-WalL  the  overview  stoplisht  graph  does  not  automatically  reflect  the  status  of  the 
various  summary  slides  when  they  are  undated. 

Fixes  to  alerting  problem  used  during  Global 

•  A  KM  fix  (business  rule)  was  to  add  an  asterisk  a  new  link  and  have  the  BWC  contact  the  AD 
when  it’s  been  read  (via  e-mail).  Also,  include  an  exclamation  point  after  a  link  that  the  AD 
thinks  really  needs  to  be  read/addressed. 

•  Suggest  the  use  of  ‘Last  update  date’  to  determine  when  pages  were  last  updated. 

-  Has  .  . .  been  updated? 

•  BW  and  K-Wall-op  monitoring  “blue  link”  and  verbal  alerting  [but  problems  exist  with 
this. . . .] 

-  I’m  looking  for  blue  text  as  an  indication  of  new  information. . .  . 

-  When  I’m  looking  for  pages,  I  don’t  use  the  overview  slide  ...  I  look  for  blue  links  and 
asterisks. 

-  We  could  use  blue  links  and  time. . . . 

-  [BW  announcing]  New  alert  on  ISR  coming  up  on  the  summary  page. 

-  TMD’s  got  something  new.  [referring  to  blue  link] 

•  Notification  (by  ADs)  via  e-mail,  chat,  phone  (to  BWC  or  K-Wall  techs),  walking  up  to  ICC 

-  e-mail  to  75-per  K-Wall  update.  .  . . 

-  chat  [in  IWS]  to  J35 —  “I’ve  updated  K-Wall  page.” 

•  Use  of  colored  text  or  pictures  to  alert  (by  ADs) 

-  Use  the  picture  area  to  alert  (with  text  “alert”). 

Feedback  to  ADs 

•  We  need  a  way  for  K-Wall  users  to  convey  desires/disseminate  information  “down”  to  anchor 
desks  and  lower  echelons. 

•  [Information  provider  in  the  same  room  as  JCC\  I  have  an  advantage  over  people  downstairs 
because  I  can  walk  over  and  see  if  they’ve  looked  at  my  links  or  not  [using  color  of  link  text]. 

•  Give  the  admirals  a  “laser”  and  check  boxes  that  they  can  check  once  they’ve  read  something. 

•  ...  there  should  be  a  feedback  loop  and  rules  for  stoplights. 

•  There  needs  to  be  automatic  feedback  that  the  page  has  been  read. 

•  A  KM  fix  (business  rule)  was  to  add  an  asterisk  a  new  link  and  have  the  BWC  contact  the  AD 
when  it’s  been  read  (via  e-mail).  Also,  include  an  exclamation  point  after  a  link  that  the  AD 
thinks  really  needs  to  be  read/addressed. 

•  There  needs  to  be  a  feedback  loop  on  the  K-Wall  from  the  BWC  to  the  AD/FA. 

•  Feedback  notification  is  required  when  something  urgent  has  been  received  by  the  CJTF. 

•  I  recommend  that  the  signal  be  cleared  once  the  CDR  views  the  status. 
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K-Wall  Access  and  Communication  Issues,  CONORS,  Training,  Business  Rules 

•  Who  should  have  access  to  the  K-Wall  summary  pages? 

•  Who  should  have  access  to  a  K-Wall? 

•  How  flexible/tailorable  should  K-Wall/SPs  be  (wrt  different  users)? 

•  How  does  the  K-Wall  promote/prevent  vertical  and  horizontal  communication? 

•  How  does  the  K-Wall  change  the  CONOPS? 

K-Wall  Access  (who,  how  and  what) 

Access  to  Summary  Pages 

•  Another  important  feature  is  that  you  can  call  pages  up  on  other  machines. 

•  There  was  a  request  by  the  BWC  for  a  CJTF  summary  page  that  the  ADs  can  access  and 
development  of  CJTF  page  for  CINC/NCA  access. 

•  [CJTF  page]  We  have  this  CJTF  page,  let’s  make  sure  we  use  it  today. 

•  The  ISR  summary  page  is  being  displayed  on  the  J2’s  console. 

•  The  COS  asked  what  was  on  M9  so  he  could  read  it  on  laptop. 

•  We  [MARFOR]  checked  the  other  pages  periodically. 

•  ...  the  individual  functional  and  component  staff  officer  still  faces  dauntins  challenses  in 
sortins  throush  reams  of  links.  I  recommend  that  next  year  we  give  players  their  own 
personalized  portal. 


Access  to  a  K-Wall 
FA  Personnel,  etc. 

•  [AD  in  the  same  room  as  the  JCC]  I  have  an  advantage  over  people  downstairs  because  I  can 
walk  over  and  see  if  they’ve  looked  at  my  links  or  not  [using  color  of  link  text]. 

•  Several  cell  personnel  (in  same  room  as  JCC)  have  come  up  to  ask  where  the  ISR  picture  (they 
can  see  on  the  K-Wall)  is  located  (i.e.,  they  are  looking  at  the  K-Wall). 

•  An  advantage  we  had  over  the  component  commanders  was  not  having  a  single  page  but 
multiple  screens. 

•  Lower  echelons  need  some  way  to  tell  what  is  currently  being  displayed  on  the  K-Wall.  An 
HTML  page  with  a  table  divided  like  the  screens,  showing  the  title  of  each  panel  content,  with 
a  link  to  that  content,  would  help. 

•  There  are  questions  on  whether  the  large  screen  displays  on  sides  of  JOC  can  be  slaved  to 
show  what  the  head  table  is  viewing  up  close  on  the  K-Wall?  These  are  not  hard  requirements 
for  this  game  but  objective  to  allow  watch  standers  to  see  what  the  head  table  is  looking  at. 

•  Since  the  K-Wall  was  only  viewable  from  JCC.  don’t  believe  the  K-Wall  was  of  much  value  [to 
the  cells  I . 


CINC/NCA 

•  There  was  concern  expressed  by  the  JCC  staff  that  the  CINC  isn’t  seeing  everything  they  can. 

•  The  same  concern  was  expressed  by  the  CINC. 

•  ....  we  noticed  we  weren’t  updating  our  seniors,  so  we  developed  a  CJTF  page. 

•  What  is  lacking  is  a  means  of  summarizing  information  on  the  K-Wall  and  passing  it  up. . . . 
What  are  the  implications,  how  are  things  unfolding? 


C-13 


•  Because  the  CINC  doesn’t  have  a  K-Wall,  the  CJTF,  via  the  BWC  needs  to  be  in  eontact  to 
keep  him  updated  [ the  connection  to  the  Coronado  was  down,  the  CINC  played  in  NWC  with 
no  K-Wall], 


Tailorability  of  K-Wall 

•  There’s  a  problem  with  [the  CINC  having]  too  much  access  to  information,  whether  it’s 
stoplights  on  pages  or  e-mail. 

•  The  tool  is  for  the  CJTF  and  the  CINC  shouldn’t  be  bugging  MARFOR,  etc.  [related  to 
Zelibor’s  comment  above] 

•  Whose  tool  is  it?  If  three  groups,  CJTF,  CINC,  NCA,  can  see  the  K-Wall  they  need  to  see 
different  information  . . .  One  K-Wall  glove  does  not  fit  all — there  needs  to  be  tailorable 
information  ...  A  green  status  alert  does  not  mean  the  same  to  everyone. 

•  Do  we  give  everyone  access  to  all  the  K-Wall  and  hope  they  access  only  the  information  they 
should  or  restrict  access  (e.g.,  NCA  should  not  see  C2PC;  information  may  be  different  if 
people  know  the  CINC,  and  everyone  else,  might  be  reading  it). 

•  The  CINC/NCA  should  have  a  customizable  version  of  the  K-Wall. 

•  [Problem  that  the  CINC  was]  more  worried  about  what  CJTF  was  doing  instead  of  what  they 
should  communicate  to  NCA 

Push/Pull:  Definitions  of  What  Goes  on  Wall/Communications  of  Commanders’  Needs 

•  There  were  lots  of  requests  for  the  JTF  to  have  control  of  alerts — ^to  be  able  to  alert  everyone 
else  (i.e.,  push  information  to  everyone  else). 

•  The  CJTF  page  is  being  created  that  is  accessible  to  the  CINC  and  component  commanders. 

•  The  CJTF  should  provide  the  CINC  with  a  picture  of  what’s  going  on  so  he  can  let  others 
know.  . .  . 

•  A  lot  of  people  are  wondering,  all  this  stuff  is  going  to  the  K-Wall,  but  how  do  they  know  what 
is  being  looked  at,  what  are  they  getting  back.  . . . 

•  Commanders  should  be  driving  K-Wall  requirements — in  the  beginning  let  the  pushers  drive 
the  wall. 

•  Sometimes  we  (CJTF  personnal)  asked  a  question  and  they  answered  it  by  pushing  to  the  wall. 
Other  times,  we  didn’t  know  the  question  to  ask  and  they  gave  us  information. 

•  We  need  to  restrict  what  goes  on  the  K-Wall.  It  should  be  a  top-down  drive  ...  the  CINC  gives 
CCIRS  .  . .  but  these  are  mission  dependent. .  . .  We  really  need  a  constant  baseline  for 
information  and  mission  related  additions. 

•  A  conversation  between  CJTF,  CINC  and  ADs  is  needed  about  what  the  stoplights  mean. 
Higher  ups  must  have  a  definition,  but  the  ADs  are  not  giving  it. 

•  The  ADs  should  define  this  .  .  .  The  CINC  wants  a  standard  definition  but  thinks  it’s  OK  if  the 
ADs  change  it .  .  .  this  doesn’t  make  sense.  Define  what  should  go  there  and  then  what  the 
colors  actually  mean. 

•  The  definition  is  a  cultural  issue  (e.g.,  different  for  a  commander  at  sea  than  a  marine). 

•  We  need  way  for  K-Wall  users  to  convey  desires/disseminate  information  “down”  to  anchor 
desks  and  lower  echelons. 

•  There’s  an  issue  of  push/pull — what  gets  put  on  the  wall?  What  the  Anchor  Desks  want  to 
push  up,  or  what  the  decision  makers  need? 

•  The  “CJTF’  page  is  vital  to  provide  two-way  communications  to  the  team. 
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•  The  K-Wall  is  great  for  passing  information. . . . 

•  The  CJTF  level  did  not  feel  comfortable  with  no  way  to  sync  with  higher  and  lower  levels — no 
way  to  push  information  or  convey  needs/plans/intentions. 

•  The  commander’ s  priorities  must  be  made  clear  to  all. 

•  We  need  to  define  what  24  hours,  48  hours,  long  term  means.  [Slaght] 

•  At  the  beginning,  information  flow  was  internal  only  -  getting  their  act  together  and  sharing 
knowledge  about  plans,  scenario,  and  system.  After  Day  1,  the  players  were  concerned  about 
sharing  information  with  their  functional  areas.  Next,  sending  information  up  the  chain.  The 
best  example  during  Globe  00  was  problems  seen  in  populating  the  K-Wall. 

•  Concern  that  stoplights  on  K-Wall  are  being  interpreted  differently  by  each  of  functional 
commanders.  What  business  rules  are  needed  from  CJTF  to  ensure  consistencies?  (What  would 
KM  role  be  here?) 

•  When  things  got  hot  and  heavy,  we  saw  the  lower  level  DMs  pushing  information  and 
requesting  for  decisions  up,  inundating  the  CJTF  with  email  vice  using  the  K-Wall  as  a  vehicle 
for  communication  actions  taken. 

Issues  Related  to  Updating  Summary  Pages 

•  If  you  really  wanted  to  do  this  right,  you’d  have  to  have  someone  updating  the  pages  full  time. 

•  It  was  a  burden  for  echelons — but  more  so  than  preparing  the  daily  brief?  Some  thought  it  was, 
others  did  not!  More  so  in  first  week,  as  people  got  used  to  the  tools,  then  there  was  more  use, 
more  “let’s  put  this  on  the  wall.” 

•  The  use  of  the  K-Wall  is  manpower  intensive  in  that  many  people  were  involved  in  its  use — 
and  the  preparation  of  the  content. 

•  The  requirement  to  post  separate  links  on  both  K-Wall  summary  pages  and  the  Battle  Plan 
(both  future  and  current)  increases  workload  and  leads  to  conflicting  information.  If  we  had 
clearly  defined  products,  we  could  just  update  products  and  links  to  them  could  be  automatic. 

•  We  believe  the  collaboration  needed  to  produce  documents  such  as  summary  slides  put  an 
unnecessary  load  on  the  IT  tools  and  the  people  involved. 

•  Overall  problem:  Task  saturation  for  the  AD  person.  You  can’t  fight  war  plan  tomorrow’s 
fight,  and  do  K-Wall  duties. 

•  Problem:  the  AD  person  was  very  involved  in  planning  the  battle  .  . .  updating  the  K-Wall  was 
a  burden  in  the  beginning.  Bottom  line:  for  a  nearly  real  time  K-Wall,  a  dedicated  person  must 
be  there.  If  not,  the  K-Wall  will  be  populated  when  the  AD  can  get  to  it.  It  will  be  populated 
with  information  that  is  not  timely,  but  a  placeholder  that  appeases  higher  ups  and  looks  good. 

•  How  do  you  capture  when  K-Wall  summaries  are  no  longer  something,  which  is  seen  as  a 
separate  task  but  instead  is  just  another  means  of  keeping  the  boss  informed. 

•  From  the  information  provider  view,  it  was  considered  a  burden — the  K-Wall  was  seen  as  a 
collateral  duty.  There  were  too  few  people  having  to  populate  the  K-Wall,  drive  the  game  with 
battle  plans,  and  also  update  the  K-Wall. 

•  Summary  page  development  and  update  is  a  full  time  job  and  is  best  manned  by  an  operator 
who  understands  the  mission  area  and  the  commander’s  part  in  the  mission.  They  have  become 
very  adept  at  updating. 

•  Well-manned  mission  areas  are  being  updated  faster  than  poorly  manned  ones. 

Horizontal  Communication 

•  There  are  problems  with  the  communication  between  the  different  functional  areas. 
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•  It’s  hard  to  integrate  stoplights  . . .  there  has  to  be  horizontal  integration  between  the  ftinctional 
areas  to  say  how  does  this  [red  alert]  impact  you. 

•  We  need  to  establish  linkages  between  functional  areas. 

•  How  do  we  make  collaboration  work  . . .  components  need  to  collaborate. 

•  The  goal  should  not  be  to  increase  manual  coordination  through  chat  rooms  but  to  increase 
shared  awareness  through  better  automation. 

Training  and  Business  Rules 

Other  tools;  TacGraph,  C2PC,  Caesar,  TAPS,  WIGS,  IWS 

How  well  did  these  tools  integrate  with  the  K-Wall? 

What  problems,  benefits  etc.  were  associated  with  tools? 


General 

•  There  are  too  many  independent  databases  . . .  they  don’t  correlate  input  from  the  war  game.  If 
input  is  made  it  should  update  everything  TAPS,  C2PC,  TacGraph,  and  K-Wall  should  be 
linked. 

•  Most  players  use  multiple  windows,  but  products  do  not  alarm  you  when  new  information 
becomes  available.  Since  the  screen  isn’t  that  bie.  it’s  difficult  to  keep  multiple  windows  open. 


C2PC 

•  C2PC  isn’t  Web-based — which  is  a  problem  since  everything  else  is. 

•  That’s  why  I  hate  C2PC,  none  of  that’s  accurate. 

•  Bring  up  C2PC  for  situational  awareness. 

•  C2PC — sood  enough  for  operational  level  play. 


TacGraph 

•  I  thought  that  TacGraph  was  a  more  detailed  picture  of  C2PC,  but  it’s  not. 

•  It  needs  to  be  integrated  with  C2PC. 

•  They’re  not  using  TacGraph  enough,  they  should  because  there’s  not  a  lot  of  space  on  a 
page  ...  a  picture  is  better  than  words. 

•  I  know  I  can  highlight  these  [objects  on  TacGraph  picture  on  M9,  i.e.,  link  to  information]  but 
there  are  too  many  things  up  there. 

•  An  observed  problem  is  that  navy  personnel  don’t  know  what  army  symbols  are — a  suggestion 
was  made  [BWC]  that  there  should  be  a  roll-over  function  in  TacGraph  that  shows  a 
description  of  a  unit. 

•  A  problem  [reported  by  AD]  is  that  the  font  can’t  be  changed  for  Title  in  the  TacGraph  file. 

•  Players  are  actually  trying  to  find  the  full  capability  of  the  systems.  They  are  trying  to  import 
data  from  C2PC  onto  TacGraph.  Both  systems  have  good  features  that  the  other  doesn’t.  They 
want  real  time  data  from  C2PC  on  the  TacGraph  chart  for  manipulation. 
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•  The  SC  summary  graphic  uses  a  C2PC  backdrop.  What  about  using  TacGraph  or  IMAT,  since 
it  can  show  more  useful  information. 

•  Clearly  tools  like  TacGraph  are  great  for  rapidly  building  a  picture  that  can  express 
information  concisely  and  rapidly  . . .  need  training,  not  on  how  to  operate  the  tool  but  how 
information  can  be  presented  . . .  visualizing  information.  . . . 

•  On  TacGraph,  the  zoom  needs  to  be  variable.  The  operator  would  like  to  be  able  to  control 
view  he/she  sees. 

•  TacGraph:  I  believe  this  is  a  fantastic  tool,  which  was  brought  in  too  late  in  the  game.  Its 
graphics  are  outstanding  and  its  bandwidth  is  conserved  when  documents  are  published  in 
JPEG  format. 

IWS 

•  IWS  generally  was  not  liked  from  all  perspectives — but  functions  offered  by  products  like  IWS 
were  desired. 

•  The  groups  that  were  on  IWS  did  use  chat  for  collaboration. 

•  The  whiteboard  used  by  some  groups-would  have  been  useful  for  ARFFOR  to  communicate 
with  the  CJTF. 


WIGS 

•  Opening  summary  pages  from  FAs  in  WIGS  caused  the  text  to  wrap  (on  the  laptop).  This  was 
fixed  when  settings  were  changed  to  a  large  font. 

•  We  need  to  get  everything  on  WIGS  instead  of  K-Wall  so  we  have  accountability  and  then 
push  it  when  it’s  ready. 

•  Navigating  using  the  [K-Wall]  channel  menu  is  inconvenient — e.g.,  getting  to  WIGS  involves 
many  steps. 

•  Alerting  on  WIGS  stopped  happening  late  in  the  game. 

•  I  used  it  as  a  reference  library  to  go  get  documents  .  . .  the  concept  is  great .  . .  people  want 
every  system  to  be  connected  to  a  centralized  database. . . . 

•  The  alerts  were  useless  because  not  everyone  had  WIGS  up  and  you  had  to  click  on  the  alert  to 
see  what  it  was  ...  If  there’s  an  alert,  I  want  the  screen  to  light  up — I  don’t  want  to  have  to 
pull  information. 

•  I  wasn’t  sure  if  I  had  current  information,  reliable  information.  I  need  confidence  that  the 
information  is  current  and  up  to  date  . .  .  people  talk  about  time  stamping  ...  we  need  source  of 
information  and  validity. 

•  I  need  to  make  sure  I’m  getting  information  from  my  echelon  . . .  and  if  I  need  more  I  can  go 
get  it. 

•  There  was  a  problem  when  you’re  linked  to  the  battle  plan  and  there’s  an  update  [doesn’t 
update  on  your  screen]  on  WIGS,  how  do  you  know  what’s  current? 

•  All  systems  should  be  available  or  accessible  via  WIGS.  It’s  very  hard  to  find  what  you  are 
looking  at. 

•  The  concept  was  sound  thought,  and  once  you  became  familiar  with  it,  it  could  be  useful.  The 
“link-map”  is  necessary  so  you  can  figure  out  how  to  use  them  and  where  they  are. 

•  WIGS.-  the  only  method  to  set  SA  from  a  distributed  site. 
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TAPS 


TAPS  was  displayed  on  K-Wall  X  %  of  time  (#sfrom  screen  captures,  log  files) 

•  There  was  a  discussion  of  using  TAPS  to  show  Blue’s  intended  actions  and  what  Red  thinks 
their  status  is  (is  this  what  it  really  represents?). 

•  We’re  going  to  take  TAPS  and  put  values  on  it,  even  though  it’s  not  working. 

•  TAPS  would  be  nice,  if  it’s  working  ...  Is  TAPS  updated  on  here? 

•  ...  that’s  [TAPS]  what  we’ve  done  .  .  .  there  are  unintended  consequences,  etc. 

•  It’s  ahead  of  its  time  . . .  Maybe  it’s  more  of  a  decision  aid.  You  don’t  use  it  to  make  decisions 
...  It  shows  general  flow,  are  we  heading  in  the  right  direction. . . . 

•  COP  should  not  exist ...  a  CO  baseline  should,  a  shared  story  ...  it  could  be  TAPS  at  the 
NCA  level. 

•  ...  a  great  idea  but  manually  intensive. 

•  It’s  a  great  idea  to  the  information  is  timelate  and  manpower  intensive;  it  should  be 
automated.  It’s  an  immature  tool — useless  now  but  maybe  good  some  day. 

•  I  don’t  know  what  this  is  telling  me! 

•  How  is  the  data  relevant  to  the  mission? 

•  This  is  a  bean  counter’s  tool. 

•  I  don’t  trust  it  because  I  don’t  understand  it. 

•  As  executed  in  the  game,  this  is  a  very  immature  tool.  It  looks  like  it  should  have  been  good, 
but  information  input  was  from  different  people  with  different  subjective  opinions.  If  you  don’t 
know  where  you  are  going,  any  road  will  get  you  there. . . . 

•  No  decision  points,  no  triggers,  no  sense  of  progress — maybe  it  should  be  a  decision  aid.  This 
is  way  ahead  if  its  time. 


CAESAR 

•  CAESAR  summary  page  is  displayed  on  M9;  then  a  link  to  CAESAR  PowerPoint  document, 
accessed  via  the  CAESAR  channel  =  single  time  CAESAR  was  displayed 

•  I  put  the  most  important  information  on  the  last  slide  (of  10) — they  never  even  looked  at  it . .  . 
you  need  to  put  the  most  important  information  on  the  first  slide. 


JFLEX 

•  JFLEX  plan  is  displayed  on  M9  (for  the  first  time).  It  was  a  link  from  JFLEX  summary  page 
(Misc  Links)  =  single  time  JFLEX  was  displayed. 
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